LLM & VLM courseBlog
230Track 9: Generative models · session 9 of 12

Few-step generation: consistency models and distillation

A diffusion model needs 20 to 100 network calls per image, and that is too slow for many products. This session shows why a better solver cannot go below a handful of steps, and how to train a network that takes big steps instead: progressive distillation halves the steps round after round, and consistency models jump from any noise level straight to the clean image. We then follow these ideas into real systems (LCM, LCM-LoRA, SDXL-Turbo, FLUX.1 schnell) and weigh quality against steps and latency.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Diffusion 2 (session 226: the probability-flow ODE, DDIM and the σ form), classifier-free guidance, basic PyTorch

What you will learn

  • Where the time of a diffusion sample goes, and why the speed-up in latency is smaller than the drop in network calls
  • Why one DDIM step gives a blurry posterior mean even with a perfect network
  • Progressive distillation: the two-steps-into-one target, the loss weighting, the v-parameterization and the cost in rounds
  • Consistency models: the consistency function, self-consistency, the built-in boundary condition, and multistep sampling
  • Consistency distillation vs consistency training, why the target network uses stop-gradient, and how iCT and sCM improved them
  • How LCM, LCM-LoRA, ADD, LADD, DMD and rectified flow turn these ideas into 1 to 4 step products
  • How to choose between quality, diversity, step count and end-to-end latency, for images and for video

Why this matters

A diffusion model makes an image by calling a large network again and again: 20 to 50 times with a good solver, and twice per step when classifier-free guidance is on. For a chat-style product, a phone, or a video model that must make every frame of a clip, that is too slow and too expensive. This session is about the methods that bring the count down to 1 to 4 network calls.

In session 226 you saw that sampling is solving an ordinary differential equation (ODE) from noise to data, and that better solvers get good results in about 10 to 25 steps. Below about 8 steps, solvers fail: the path from noise to data is curved, and a few big straight steps miss it. The fix is not a better solver; it is a new network that is trained to take big steps. There are two classic ways to train it, and they are the core of this session:

  • Progressive distillation (Salimans and Ho, 2022): teach a student to do in one step what the teacher does in two. Repeat, halving the step count each round.
  • Consistency models (Song et al., 2023): teach a network to jump from any point on the noise-to-data path straight to the end of that path.

Then we look at how these ideas scaled to text-to-image products (Latent Consistency Models, LCM-LoRA, SDXL-Turbo, FLUX.1 schnell), at the adversarial and distribution-matching methods that won the 1-step race, and at the honest trade-off between quality, step count, diversity and latency.

The link to LLMs is direct. In session 59 a small student copies a big teacher’s outputs. Here the student copies the teacher’s sampler: many small steps become a few large ones. It is the same economics as session 128: the expensive model is called fewer times per output.

Where the time goes

Let \(N\) be the number of sampling steps. Each step calls the denoising network (a U-Net or a diffusion transformer) once, or twice with classifier-free guidance (CFG), because CFG runs the model with and without the prompt (session 227). Around the loop there is fixed work: the text encoder runs once, and for a latent model the VAE decoder turns the final latent into pixels once. So:

\[ T_{\text{image}} = T_{\text{fixed}} + N \cdot c \cdot t_{\text{net}}, \qquad \text{NFE} = N \cdot c \tag{230.1} \]
Read the symbols
\(N\)
sampling steps
\(c\)
network calls per step: 2 with CFG, 1 without (or when guidance is built into the model)
\(t_{\text{net}}\)
time of one network call; for a large transformer at 1024 × 1024 this is often tens of milliseconds or more on one GPU [Likely]
\(T_{\text{fixed}}\)
text encoding, VAE decoding, safety checks and other work done once per image
NFE
number of function evaluations, the standard cost unit for samplers

With 50 steps and CFG, NFE = 100. A 4-step model with guidance built in has NFE = 4: 25 times fewer calls. But the speed-up in wall-clock time is smaller, because \(T_{\text{fixed}}\) does not shrink. This is Amdahl’s law: once the loop is short, the decoder and text encoder dominate. Figure 1 lets you set your own numbers. The defaults (50 ms per call, 100 ms fixed) are only an example, not a measurement; measure your own model as in session 123.

network callsnetwork calls, your settingfixed time (text encoder, decoder)
Figure 1. Formula (230.1) as a bar chart. The first four rows use your \(t_{\text{net}}\) and \(T_{\text{fixed}}\) with fixed step counts; the last row is your setting. All numbers are illustrative inputs, not benchmarks.
How to read Figure 1

At the defaults the teacher needs 100 calls and 5,100 ms; a 4-step model without CFG needs 300 ms (17 times faster, not 25, because of the 100 ms fixed part); a 1-step model needs 150 ms, two thirds of which is fixed time. Move the fixed time to 0 to see the pure NFE ratio. This is why few-step products also use a small or distilled VAE decoder and cache the text embeddings. [Likely]

Why a better solver is not enough

Recall the DDIM step from session 226 in the \(\sigma\) form, where a noisy sample is \(\mathbf{x}_\sigma = \mathbf{x}_0 + \sigma \varepsilon\). The denoiser \(D(\mathbf{x}; \sigma)\) is the network’s best guess of the clean sample. One step from noise level \(a\) to a lower level \(b\) is:

\[ \mathbf{x}_b = D(\mathbf{x}_a; a) + \frac{b}{a} \big( \mathbf{x}_a - D(\mathbf{x}_a; a) \big) \tag{230.2} \]

In words: jump to the clean guess, then add back a fraction \(b/a\) of the noise that was removed. If you go to \(b = 0\) in one step, the output is just \(D(\mathbf{x}_a; a)\). But the best possible denoiser is the average of all clean images that could have produced \(\mathbf{x}_a\): \(D(\mathbf{x}; \sigma) = \E[\mathbf{x}_0 \mid \mathbf{x}_\sigma = \mathbf{x}]\). At high noise, almost every image could have produced \(\mathbf{x}_a\), so the average is a grey blur. Even a perfect network gives a blurry image in one DDIM step. It is not a training problem; it is what the denoiser is defined to be.

On the 1-D toy of session 226 (a mixture of three Gaussians with means −2, 0.5 and 2.5, and its exact denoiser) we measure sample quality with the 1-D Wasserstein-1 distance \(W_1\): the average distance between the sorted samples and the true data quantiles (0 is perfect). DDIM with the perfect denoiser and the EDM noise schedule (\(\sigma_{\max} = 80\), \(\sigma_{\min} = 0.002\), \(\rho = 7\)) gives \(W_1 = 1.41\) at 1 step, 0.92 at 4 steps, 0.38 at 8, 0.18 at 16 and 0.042 at 64. Each halving of steps roughly doubles the error, until the samples collapse to the middle. Every method below beats this curve by training the network for big steps.

Progressive distillation: halve the steps again and again

Start with a trained teacher and a deterministic sampler with \(N\) steps, such as DDIM. Make a student network, initialized as a copy of the teacher. Train the student so that one of its steps lands where two teacher steps land. Now the student samples in \(N/2\) steps with nearly the teacher’s quality. Make the student the new teacher and repeat. After \(k\) rounds, the sampler has \(N / 2^k\) steps.

Why is one round easy to learn? Because two adjacent small steps are almost one straight step: the student has to learn only a small correction. Learning a 1000-step to 1-step map in one go is much harder; the paper’s point is that 2 to 1 is always a gentle task.

The training target

The student is a denoiser \(D_\eta\), used inside the same DDIM update (230.2). Suppose the teacher goes from \(\mathbf{x}_a\) to an intermediate level and then to \(\mathbf{x}_b''\), in two DDIM steps. Which clean guess \(\tilde{\mathbf{x}}\) would make one DDIM step from \(a\) land exactly at \(\mathbf{x}_b''\)? Put \(\tilde{\mathbf{x}}\) in place of \(D\) in (230.2) and solve \(\mathbf{x}_b'' = \tilde{\mathbf{x}} + \tfrac{b}{a}(\mathbf{x}_a - \tilde{\mathbf{x}})\):

\[ \tilde{\mathbf{x}} = \frac{a\, \mathbf{x}_b'' - b\, \mathbf{x}_a}{a - b} \qquad \Big(\text{paper form: } \tilde{\mathbf{x}} = \frac{\mathbf{z}_{t''} - (\sigma_{t''}/\sigma_t)\, \mathbf{z}_t}{\alpha_{t''} - (\sigma_{t''}/\sigma_t)\, \alpha_t}\Big) \tag{230.3} \]

The second form is the one in the paper, written for the variance-preserving process \(\mathbf{z}_t = \alpha_t \mathbf{x} + \sigma_t \varepsilon\); with \(\alpha = 1\) it is the first form. [Certain] The student is trained with a weighted squared error to this target:

\[ \mathcal{L}(\eta) = \E_{\mathbf{x}_0,\, \varepsilon,\, a} \Big[ w(a)\, \big\lVert D_\eta(\mathbf{x}_a; a) - \tilde{\mathbf{x}} \big\rVert^2 \Big], \qquad w = \max(\text{SNR}, 1) \tag{230.4} \]
Read the symbols
\(\mathbf{x}_a\)
a training image with noise of level \(a\) added: \(\mathbf{x}_0 + a\,\varepsilon\); \(a\) is one of the student’s noise levels
\(\mathbf{x}_b''\)
where two DDIM steps of the (frozen) teacher end, starting from \(\mathbf{x}_a\)
\(\tilde{\mathbf{x}}\)
the target: the clean guess that makes one student step match the two teacher steps; it is a regression target, computed without gradients
SNR
signal-to-noise ratio, \(\alpha^2/\sigma^2\) (here \(1/a^2\)); “truncated SNR” weighting \(\max(\text{SNR}, 1)\) is one of two weightings proposed in the paper (the other is \(\text{SNR} + 1\)); plain SNR weighting gives zero weight at the noisiest level, which is exactly where the 1-step student must work
Worked example: one distillation target, with the exact teacher

On the 1-D toy, take \(\mathbf{x}_a = 1.2\) at \(a = 2\). The teacher uses two DDIM steps, \(2 \to 1 \to 0.5\).

Step 1: \(D(1.2; 2) = 0.635\) (the posterior gives the three modes weights 0.146, 0.615 and 0.239, so the guess is an average). Then \(\mathbf{x}_1 = 0.635 + \tfrac12 (1.2 - 0.635) = 0.918\).

Step 2: \(D(0.918; 1) = 0.767\), then \(\mathbf{x}_b'' = 0.767 + \tfrac12 (0.918 - 0.767) = 0.843\).

Target (230.3): \(\tilde{\mathbf{x}} = (2 \times 0.843 - 0.5 \times 1.2) / 1.5 = 0.723\). Check: one DDIM step with this guess gives \(0.723 + \tfrac14 (1.2 - 0.723) = 0.843\), the two-step result. One plain teacher step \(2 \to 0.5\) would land at 0.776 instead.

Compare the three “clean guesses”: the denoiser says 0.635, the target says 0.723, and the true end of the ODE path through \((1.2, \sigma = 2)\) is 0.716. The target is not the average clean image any more; it already points toward the end of the path. Each round pushes this further.

Algorithm 1: progressive distillation (Salimans and Ho, 2022)
  1. input: trained teacher \(D_\theta\); its sampler with \(N\) steps on noise levels \(\sigma_0 \gt \sigma_1 \gt \dots \gt \sigma_N = 0\)
  2. while \(N \gt\) target steps:
  3.   student \(D_\eta \leftarrow\) copy of \(D_\theta\) // warm start: the student is already a good denoiser
  4.   repeat (for example 50k updates):
  5.     pick a data sample \(\mathbf{x}_0\), a student level \(a = \sigma_{2j}\), set \(b = \sigma_{2j+2}\); \(\mathbf{x}_a \leftarrow \mathbf{x}_0 + a\,\varepsilon\)
  6.     \(\mathbf{x}_b'' \leftarrow\) two teacher DDIM steps \(a \to \sigma_{2j+1} \to b\) // 2 teacher calls, no gradients
  7.     \(\tilde{\mathbf{x}} \leftarrow (a\,\mathbf{x}_b'' - b\,\mathbf{x}_a)/(a - b)\) (if \(b = 0\): \(\tilde{\mathbf{x}} \leftarrow \mathbf{x}_b''\)) // (230.3)
  8.     gradient step on \(w(a)\,\lVert D_\eta(\mathbf{x}_a; a) - \tilde{\mathbf{x}} \rVert^2\) // (230.4)
  9.   \(D_\theta \leftarrow D_\eta\); keep every other noise level; \(N \leftarrow N/2\) // the student becomes the teacher
  10. return \(D_\theta\) and its \(N\)-step grid

Figure 2 runs this on the 1-D toy. The teacher is 64 DDIM steps with the exact denoiser. Each student step is a lookup table with \(K\) points (placed at quantiles of the noisy data at that level) and straight lines between them; \(K\) plays the role of network capacity. The student is fitted to the exact two-step target at its points.

student capacity \(K\):
distilled studentplain DDIM, perfect denoiser, same steps (dashed)true density
Figure 2. Six rounds of progressive distillation, 64 → 32 → 16 → 8 → 4 → 2 → 1 steps, on the 1-D mixture. Each round the student learns to cover two of the previous model’s steps with one. The error is \(W_1\) from 400 evenly spaced noise quantiles, so there is no sampling noise.
How to read Figure 2

Step through the rounds with \(K = 64\). The student’s error stays almost flat: 0.0416 (teacher), 0.0445 at 32 steps, 0.0474 at 1 step. Plain DDIM on the same noise levels grows from 0.084 to 1.41: at 1 step the distilled student is about 30 times better. Now pick \(K = 8\): the first round already loses quality (0.114) and every further round adds a little (0.143 at 1 step). With \(K = 256\) the student is essentially perfect (0.041 at every round). Lesson: distillation preserves the teacher’s quality only as far as the student can represent the sharper map; errors from each round are inherited by the next.

Details that matter in practice

  • Parameterization. With \(\varepsilon\)-prediction, the clean guess is \(\hat{\mathbf{x}}_0 = (\mathbf{z}_t - \sigma_t \hat\varepsilon)/\alpha_t\). At the noisiest level \(\alpha_t \to 0\), so tiny errors in \(\hat\varepsilon\) are blown up. A 1-step student lives exactly there. The paper therefore predicts \(\mathbf{x}\) directly, or both \(\mathbf{x}\) and \(\varepsilon\), or v-prediction, \(\mathbf{v} = \alpha_t \varepsilon - \sigma_t \mathbf{x}\), with \(\hat{\mathbf{x}} = \alpha_t \mathbf{z}_t - \sigma_t \hat{\mathbf{v}}\). [Certain] v-prediction later became common in many diffusion models. [Likely]
  • Cost. Each round is 50k updates in the paper (100k for the rounds that reach 2 and 1 steps). CIFAR-10 starts from an 8192-step teacher; the larger datasets start from 1024 steps. The authors argue the whole procedure costs no more than training the original model. [Certain] From 8192 to 4 steps is \(\log_2(8192/4) = 11\) rounds.
  • Results. On CIFAR-10 the distilled models reach FID 9.12 at 1 step, 4.51 at 2, 3.00 at 4 and 2.57 at 8 steps. [Certain] (FID, the Fréchet Inception Distance, compares feature statistics of generated and real images; lower is better.)
  • Guided models. Text-to-image models use CFG, so a plain teacher needs 2 calls per step. Meng et al. (2023) first distill the guided output (with the guidance scale as an input) into one network, then distill progressively. They report 1 to 4 denoising steps for latent models such as Stable Diffusion and at least 10 times faster inference than existing methods on ImageNet 256 × 256 and LAION. [Certain]

Consistency models: jump from any noise level to the clean image

Progressive distillation needs \(\log_2 N\) rounds and gives one fixed grid of steps. Consistency models (Song, Dhariwal, Chen and Sutskever, 2023) learn the end point of the ODE directly, in one training run.

The probability-flow ODE of session 226 is deterministic: every starting noise \(\mathbf{x}_T\) traces one path (a trajectory) down to one clean sample. So there is a function that takes any point on a trajectory, at any noise level, and returns that trajectory’s end point. That is the consistency function:

\[ f(\mathbf{x}_t, t) = \mathbf{x}_\epsilon \ \ \text{for all } t \in [\epsilon, T] \text{ on one trajectory}, \qquad f(\mathbf{x}_t, t) = f(\mathbf{x}_{t'}, t'), \qquad f(\mathbf{x}, \epsilon) = \mathbf{x} \tag{230.5} \]
Read the symbols
\(t\)
the noise level (the paper writes \(t\) for what we called \(\sigma\); in the EDM setup they are equal)
\(\epsilon, T\)
the smallest and largest noise levels; the paper uses \(\epsilon = 0.002\) and \(T = 80\) [Certain]
\(f(\mathbf{x}_t, t) = f(\mathbf{x}_{t'}, t')\)
self-consistency: two points on the same trajectory must give the same answer
\(f(\mathbf{x}, \epsilon) = \mathbf{x}\)
the boundary condition: at (almost) zero noise, the answer is the input itself

A network \(f_\theta\) that learns this function is a consistency model. Sampling is one call: draw \(\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, T^2 I)\) and output \(f_\theta(\mathbf{x}_T, T)\).

The difference from the denoiser is the whole point. The denoiser \(D(\mathbf{x}; \sigma)\) returns the average of all clean images consistent with \(\mathbf{x}\). The consistency function returns one specific clean image: the one the ODE would reach. At low noise the two agree. At high noise the average is a blur, but the trajectory end is a sharp, real-looking sample. Figure 3 shows both on the exact trajectories of the 1-D toy. In 1-D the probability-flow ODE never lets two paths cross, so it keeps the order of points: the path through the \(u\)-quantile of the noisy distribution ends at the \(u\)-quantile of the data. That gives the exact consistency function for the toy:

\[ f(x, \sigma) = Q_0\big( F_\sigma(x) \big) \qquad \text{(1-D only)} \tag{230.6} \]

where \(F_\sigma\) is the cumulative distribution function (CDF) of the noisy data at level \(\sigma\) and \(Q_0\) is the quantile function (inverse CDF) of the clean data.

selected trajectory and point \(\mathbf{x}_\sigma\) (faded: points already visited)consistency function \(f\): the end of the pathdenoiser \(D\): the average guess
  1. pick one trajectory: the \(u\)-quantile path; walk \(\sigma\) from \(T\) down to \(\epsilon\)
  2. \(x_\sigma \leftarrow Q_\sigma(u)\) // the point of this path at level \(\sigma\)
  3. \(f \leftarrow Q_0(F_\sigma(x_\sigma))\) // (230.6): the end of the path
  4. \(D \leftarrow \E[x_0 \mid x_\sigma]\) // the denoiser: an average
  5. \(F \leftarrow (f - c_{\text{skip}}(\sigma)\, x_\sigma) / c_{\text{out}}(\sigma)\) // what the network must output, (230.7)
Figure 3. Nineteen exact ODE trajectories of the 1-D mixture, from data (left, \(\sigma = 0.002\)) to noise (right, \(\sigma = 80\)). The vertical axis is \(x/\sqrt{1 + \sigma^2}\), so that all noise levels fit on one chart. The trajectories bunch into three groups at the left: the three modes. Press Play to walk down one trajectory: every visited point sends its arrow to the same end point, while the denoiser’s guess moves.
How to read Figure 3

Press Play (or Step). The animation visits nine noise levels on the 67% trajectory, from \(\sigma = 80\) down to \(\sigma = 0.002\). At each level it runs the four lines of the box: it finds the point \(x_\sigma\), computes \(f\) with (230.6), computes the denoiser \(D\), and computes what the network \(F\) must output. The table on the right fills up: the \(f\) column is 0.723 in every row (self-consistency), while \(D\) is 0.041 at \(\sigma = 80\), 0.645 at \(\sigma = 2\), 0.824 at \(\sigma = 1\) and 0.723 only at the end. The \(F\) column falls from 1.443 to 0.102 as \(c_{\text{skip}}\) grows from 0 to 1. You can also use the sliders at any time. Keep the trajectory fixed and move \(\sigma\): the point slides along the path and the round dot at the left never moves. The diamond (the denoiser) moves: 0.648 at \(\sigma = 0.5\), 0.645 at \(\sigma = 2\), 0.151 at \(\sigma = 10\), sliding toward the overall data mean (0.025) as the noise grows. Move \(\sigma\) to the left end: \(c_{\text{skip}} = 1\), \(c_{\text{out}} = 0\) and the output is the input (the boundary condition). Move a trajectory near 50%: it sits between two modes at high noise and only decides late (below \(\sigma \approx 1\)) where to go.

Building the boundary condition into the network

The boundary condition \(f(\mathbf{x}, \epsilon) = \mathbf{x}\) is not learned; it is built in with a skip connection, in the style of EDM (Karras et al., 2022):

\[ f_\theta(\mathbf{x}, t) = c_{\text{skip}}(t)\, \mathbf{x} + c_{\text{out}}(t)\, F_\theta(\mathbf{x}, t), \quad c_{\text{skip}}(t) = \frac{\sigma_{\text{data}}^2}{(t - \epsilon)^2 + \sigma_{\text{data}}^2}, \quad c_{\text{out}}(t) = \frac{\sigma_{\text{data}} (t - \epsilon)}{\sqrt{\sigma_{\text{data}}^2 + t^2}} \tag{230.7} \]
Read the symbols
\(F_\theta\)
the actual neural network (a U-Net or a transformer), free of constraints
\(\sigma_{\text{data}}\)
the standard deviation of the data; 0.5 for images scaled to \([-1, 1]\) in the paper [Certain]
\(c_{\text{skip}}, c_{\text{out}}\)
mixing weights: at \(t = \epsilon\) they are exactly 1 and 0, so \(f_\theta(\mathbf{x}, \epsilon) = \mathbf{x}\) for any weights \(\theta\); at \(t = 2\) they are 0.0589 and 0.4846; at \(t = 80\) they are about 0.00004 and 0.5, so the output comes almost only from the network

Consistency distillation (CD)

How do we train \(f_\theta\) without knowing the trajectories? Use a pretrained diffusion model as a teacher, and enforce self-consistency on neighbouring points only. Split \([\epsilon, T]\) into \(N\) levels \(t_1 = \epsilon \lt t_2 \lt \dots \lt t_N = T\). Take a training image, add noise of level \(t_{n+1}\), run one teacher ODE step down to \(t_n\), and ask the model to give the same answer at both points:

\[ \mathcal{L}_{\text{CD}} = \E\Big[ \lambda(t_n)\, d\big( f_\theta(\mathbf{x}_{t_{n+1}}, t_{n+1}),\ f_{\theta^-}(\hat{\mathbf{x}}_{t_n}, t_n) \big) \Big], \qquad \hat{\mathbf{x}}_{t_n} = \mathbf{x}_{t_{n+1}} + (t_n - t_{n+1})\, \Phi(\mathbf{x}_{t_{n+1}}, t_{n+1}) \tag{230.8} \]
Read the symbols
\(\mathbf{x}_{t_{n+1}}\)
\(\mathbf{x}_0 + t_{n+1} \mathbf{z}\) for a training image \(\mathbf{x}_0\) and \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\)
\(\Phi\)
the teacher ODE solver’s update direction; for Euler it is \((\mathbf{x} - D(\mathbf{x}; t))/t\), the DDIM step (230.2); the paper’s best CIFAR-10 setting used Heun (2 teacher calls) with \(N = 18\) [Certain]
\(d\)
a distance: squared L2, L1, or LPIPS (a learned perceptual distance); LPIPS was best on CIFAR-10 in the paper [Certain]
\(\lambda(t_n)\)
a weight; \(\lambda \equiv 1\) worked well [Certain]
\(\theta^-\)
the target network: a slowly moving copy, \(\theta^- \leftarrow \text{stopgrad}(\mu\, \theta^- + (1 - \mu)\, \theta)\); no gradient flows through it

Why does this work? At \(n = 1\) the target is \(f(\hat{\mathbf{x}}_{\epsilon}, \epsilon) = \hat{\mathbf{x}}_\epsilon\), which is exact by the boundary condition. Once the model is right at level \(t_n\), the loss makes it right at \(t_{n+1}\), and so on up to \(T\). The correct answer is passed up the ladder of noise levels, one rung per teacher step. This is bootstrapping, the same idea as temporal-difference learning in RL, where the target is the model’s own (slowly updated) prediction at the next state (session 65).

The same argument shows the limit: CD learns the end point of the teacher’s discretized ODE, not of the true ODE. If the model is perfect, a 1-step consistency sample is exactly as good as running the teacher solver for all \(N - 1\) steps. The paper bounds the error by the solver’s local error. [Certain] So \(N\) and the teacher solver set a floor on quality, just as in Figure 2 the 64-step teacher set the floor.

Algorithm 2: consistency distillation (Song et al., 2023)
  1. input: teacher denoiser \(D\); ODE solver \(\Phi\) (Euler or Heun); levels \(t_1 = \epsilon \lt \dots \lt t_N = T\); EMA rate \(\mu\)
  2. \(\theta^- \leftarrow \theta\)
  3. repeat:
  4.   \(\mathbf{x}_0 \sim\) data; \(n \sim \text{Uniform}\{1, \dots, N - 1\}\); \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\); \(\mathbf{x} \leftarrow \mathbf{x}_0 + t_{n+1} \mathbf{z}\)
  5.   \(\hat{\mathbf{x}} \leftarrow\) one solver step from \((\mathbf{x}, t_{n+1})\) to \(t_n\) with the teacher // 1 or 2 teacher calls, no gradients
  6.   \(\mathcal{L} \leftarrow \lambda(t_n)\, d\big(f_\theta(\mathbf{x}, t_{n+1}),\ f_{\theta^-}(\hat{\mathbf{x}}, t_n)\big)\) // (230.8), \(f\) as in (230.7)
  7.   \(\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}\)
  8.   \(\theta^- \leftarrow \text{stopgrad}(\mu\, \theta^- + (1 - \mu)\, \theta)\) // slow target

Figure 4 runs Algorithm 2 for real, in your browser, on the 1-D toy. The teacher is the exact denoiser of the mixture with one Euler or Heun step. The model \(F_\theta\) is tiny: at each level \(t_2, \dots, t_N\) it is a table of 16 numbers (at 16 quantiles of the noisy data, straight lines between them), plugged into (230.7). All tables start at zero, so at first \(f_\theta \approx c_{\text{skip}}\, x\), which is almost 0 at high noise. Training is plain SGD on one sample per step, with weight \(\lambda = 1/c_{\text{out}}(t_{n+1})^2\) and learning rate \(\eta = 0.2\), and a target network updated by EMA. Watch two things: the boxed formulas with the live numbers of the current sample, and the per-level error bars, which fall from left to right as the correct answer is passed up the ladder.

clean sample \(x_0\)\(x\) at \(t_{n+1}\) and online output \(a = f_\theta(x, t_{n+1})\)teacher step to \(\hat{x}\) at \(t_n\) and target \(b = f_{\theta^-}(\hat{x}, t_n)\)the gap \(a - b\)
  1. \(x_0 \sim\) data; \(n \sim \text{Uniform}\{1, \dots, N-1\}\); \(z \sim \mathcal{N}(0, 1)\)
  2. \(x \leftarrow x_0 + t_{n+1}\, z\) // noisy point
  3. \(\hat{x} \leftarrow x + (t_n - t_{n+1})\, \Phi(x, t_{n+1})\) // one teacher step, no gradient
  4. \(a \leftarrow f_\theta(x, t_{n+1}) = c_{\text{skip}} x + c_{\text{out}} F_\theta(x, t_{n+1})\) // online
  5. \(b \leftarrow f_{\theta^-}(\hat{x}, t_n)\) // target network; \(b = \hat{x}\) when \(n = 1\)
  6. \(\mathcal{L} \leftarrow \lambda\, (a - b)^2\), \(\lambda = 1/c_{\text{out}}(t_{n+1})^2\) // (230.8)
  7. \(\theta \leftarrow \theta - \eta\, \nabla_\theta \mathcal{L}\) // only 2 table entries move
  8. \(\theta^- \leftarrow \mu\, \theta^- + (1 - \mu)\, \theta\) // slow target
Figure 4. Consistency distillation trained live (seeded, so every run is the same). Top: the ladder of noise levels \(t_1 = \epsilon \lt \dots \lt t_N = T\) (vertical lines, EDM spacing) with faint exact trajectories; the current sample is drawn with its teacher step and the two network outputs at the left edge. Bottom left: mean error of \(f_\theta\) at each level against the teacher chain (the function CD converges to). Bottom right: one-step error \(W_1\) of \(f_\theta(\mathbf{x}_T, T)\) on 400 noise quantiles; the dashed line is the teacher chain itself.
How to read Figure 4

Press Step a few times at slow speed: each press runs one line of the box, and the right panel shows that line’s formula with the numbers of this sample. Then raise the speed: Play now runs many SGD steps per frame. With the defaults (\(N = 10\), Heun, \(\mu = 0.9\)) the one-step error is still 1.37 after 1,000 steps (it starts at 1.44), because the top levels have not learned anything yet: their targets are still wrong. Meanwhile the error bars of the low levels fall first, then the next ones, from left to right. At 4,000 steps \(W_1\) is 0.14, and later it settles near 0.17 to 0.18, close to the dashed teacher-chain value 0.167: CD cannot beat the solver it distills. Try \(N = 18\) with the Euler teacher: more rungs, so the information needs longer to climb (\(W_1\) is still 0.77 at 8,000 steps), but the floor is lower (0.155). Try \(\mu = 0.99\): the slower target delays learning (1.10 instead of 0.80 at 2,000 steps).

Consistency training (CT): no teacher at all

The teacher’s only job in (230.8) is to give a point \(\hat{\mathbf{x}}_{t_n}\) on the same trajectory as \(\mathbf{x}_{t_{n+1}}\). Consistency training replaces it with a cheap estimate: use the same training image and the same noise \(\mathbf{z}\) at both levels:

\[ \mathcal{L}_{\text{CT}} = \E\Big[ \lambda(t_n)\, d\big( f_\theta(\mathbf{x}_0 + t_{n+1} \mathbf{z},\ t_{n+1}),\ f_{\theta^-}(\mathbf{x}_0 + t_n \mathbf{z},\ t_n) \big) \Big] \tag{230.9} \]

This is justified because the score has an unbiased estimate from a single training pair, \(\nabla \log p_t(\mathbf{x}_t) = -\E[(\mathbf{x}_t - \mathbf{x}_0)/t^2 \mid \mathbf{x}_t]\); as \(N\) grows the CT loss matches the CD loss with an Euler teacher. [Certain] CT turns consistency models into a stand-alone family of generative models, not just a distillation trick. In the first paper CT was clearly worse than CD: one-step FID 8.70 vs 3.55 on CIFAR-10, and 13.0 vs 6.20 on ImageNet 64 × 64. [Certain]

Later work closed the gap. Improved consistency training (iCT; Song and Dhariwal, 2023) removed the EMA from the teacher network (they found it was a flaw in the theory), replaced LPIPS with a Pseudo-Huber loss, used a lognormal noise schedule, and doubled \(N\) at set intervals during training; it reports one-step FID 2.51 on CIFAR-10 and 3.25 on ImageNet 64 × 64, and 2.24 and 2.77 with two steps. [Certain] sCM (Lu and Song, 2024) makes training work in continuous time (no discrete \(N\)), scales to 1.5B parameters, and reports two-step FID 2.06 on CIFAR-10, 1.48 on ImageNet 64 × 64 and 1.88 on ImageNet 512 × 512, within 10% of the best diffusion models. [Certain]

Multistep consistency sampling

One step gives one sample. If you have budget for more, re-noise and denoise again:

\[ \mathbf{x} \leftarrow f_\theta(\mathbf{x}_T, T); \qquad \text{for } \tau_1 \gt \tau_2 \gt \dots:\quad \hat{\mathbf{x}}_{\tau_k} = \mathbf{x} + \sqrt{\tau_k^2 - \epsilon^2}\; \mathbf{z}_k, \quad \mathbf{x} \leftarrow f_\theta(\hat{\mathbf{x}}_{\tau_k}, \tau_k) \tag{230.10} \]

The extra noise is fresh, so this is not an ODE solver; it is “jump to the end, step back up to a middle noise level, jump to the end again”. The second jump starts from a lower noise level, where the model is more accurate, and it can fix details of the first sample. The paper chooses the \(\tau_k\) greedily with a search on FID. [Certain] On CIFAR-10, CD improves from FID 3.55 (1 step) to 2.93 (2 steps); on ImageNet 64 × 64 from 6.20 to 4.70. Progressive distillation on the same tables: 8.34 and 5.58 on CIFAR-10, 15.39 and 8.95 on ImageNet 64 × 64. [Certain]

Figure 5 animates (230.10) on the 1-D toy. The model is a fully trained CD model (the fixed point of (230.8), the same as in Figure 6). 400 starting noises (the 400 quantiles of \(\mathcal{N}(0, T^2)\)) jump to the clean side in one call; then all of them are re-noised with fresh noise and jump again. The re-noise levels are \(\tau = 2, 0.6, 0.15\), each rounded to the nearest level of the model’s grid. The fresh noise is drawn with a fixed seed and is stratified (a shuffled set of normal quantiles), so that 400 points give a stable error.

samples (200 of the 400 drawn)one tracked sample (numbers in the box)true data density (left)
  1. \(\mathbf{x}_T \sim \mathcal{N}(0, T^2)\) // 400 starting noises
  2. \(\mathbf{x} \leftarrow f_\theta(\mathbf{x}_T, T)\) // one jump: this is 1-step sampling
  3. for \(k = 1, \dots, K - 1\):
  4. \(\mathbf{z}_k \sim \mathcal{N}(0, 1)\) // fresh noise
  5. \(\hat{\mathbf{x}} \leftarrow \mathbf{x} + \sqrt{\tau_k^2 - \epsilon^2}\, \mathbf{z}_k\) // step back up to \(\tau_k\)
  6. \(\mathbf{x} \leftarrow f_\theta(\hat{\mathbf{x}}, \tau_k)\) // jump again, from lower noise
  7. return \(\mathbf{x}\) // \(K\) network calls in total
Figure 5. Multistep consistency sampling (230.10), animated. Same axes as Figure 3: log noise level to the right, \(x/\sqrt{1+\sigma^2}\) up. Each jump is one network call from the current noise level to \(\epsilon\); each re-noise moves every sample from \(\epsilon\) up to \(\tau_k\) with fresh noise. The histogram at the left shows the current clean samples against the true density, and the readout gives \(W_1\) after each call.
How to read Figure 5

Press Play. All samples start at the right (\(\sigma = 80\)) and jump to the left in one call. With \(N = 10\) and the Heun teacher, this one-step sample has \(W_1 = 0.167\): the modes are there but a bit too wide. Then the samples spread out again to \(\tau_1 = 1.50\) (the grid level nearest to 2) and jump back: \(W_1 = 0.128\). Two more rounds, at \(\tau = 0.47\) and \(0.117\), give 0.084 and 0.073. Each extra call costs one more network evaluation and fixes part of the error that the first long jump left. The tracked sample shows the numbers of (230.10) at each line. With \(N = 18\) and Heun, one step is already good (0.040). The second step is worse (0.065): the jump back from \(\tau = 1.92\) brings back some error. Only 3 or 4 steps go below one step (0.036 and 0.032). Extra steps help most when one step is weak, and \(\tau\) must be chosen with care (the paper searches for it).

Figure 6 runs consistency distillation on the 1-D toy. The model at each level is again a lookup table (96 points), trained level by level so that (230.8) is exactly zero at its points, which is the fixed point that CD training moves toward.

1-step error, Euler teacher1-step error, Heun teacheryour setting (ring: 2-step)
Figure 6. Consistency distillation on the 1-D mixture with the exact teacher denoiser and EDM levels (\(\epsilon = 0.002\), \(T = 80\), \(\rho = 7\)). Two-step sampling re-noises to the level of the grid nearest to \(\tau\) and applies \(f\) again (230.10); its output distribution is computed exactly, without random samples.
How to read Figure 6

At the default (Heun, \(N = 18\), the paper’s CIFAR-10 setting), one step gives \(W_1 = 0.0402\), about 35 times better than one DDIM step (1.41). Switch to 2-step sampling: with \(\tau \approx 1.09\) the error drops to 0.0237. Switch the teacher to Euler: one step gives 0.153, close to what a 17-step DDIM sampler gives, as the theory says. Now slide \(N\): the error falls about as \(1/N\) with Euler and \(1/N^2\) with Heun (Euler 0.071 and Heun 0.011 at \(N = 40\)). But at \(N = 3\) Heun is far worse (5.99 vs 0.78): its second evaluation lands at a point where the slope says little about a step that long. Two-step sampling helps most when one step is weak; try \(\tau\) near 1 to 1.5.

From toy models to products

Latent Consistency Models (LCM) and LCM-LoRA

LCM (Luo et al., 2023) applies consistency distillation to Stable Diffusion in its latent space (session 228). Three changes make it practical:

  • Guidance inside the ODE. The teacher’s direction uses the CFG-combined noise prediction, so the student learns the guided ODE and needs no second call at sampling time:
\[ \tilde\varepsilon_\theta(\mathbf{z}_t, \omega, \mathbf{c}, t) = (1 + \omega)\, \varepsilon_\theta(\mathbf{z}_t, \mathbf{c}, t) - \omega\, \varepsilon_\theta(\mathbf{z}_t, \varnothing, t) \tag{230.11} \]
Read the symbols
\(\mathbf{c}, \varnothing\)
the text condition and the empty prompt
\(\omega\)
the guidance scale; during training it is drawn from \([2, 14]\) and given to the student as an extra input (with a zero-initialized embedding), so it can be chosen at sampling time; the tables use \(\omega = 8\) [Certain]
  • Skipping steps. Instead of enforcing consistency between adjacent levels of the 1000-step schedule, LCM enforces it between \(t_{n+k}\) and \(t_n\) with \(k = 20\), using a DDIM teacher step over that gap. This makes the bootstrapping chain 50 links long instead of 1000, which speeds up training. [Certain]
  • Cheap training. The authors report 32 A100 GPU hours for a 768 × 768, 2-to-4-step model. [Certain]

LCM-LoRA (Luo et al., 2023) trains the consistency distillation as a LoRA adapter (session 57) for SD-v1.5, SSD-1B and SDXL. The authors report that the same adapter can be plugged into other fine-tuned versions of the base model, without more training, and call it a “universal acceleration module”. [Certain] This matters for products: one small file turns a whole ecosystem of style fine-tunes into 4-step models.

Adversarial and distribution-matching distillation

Trajectory methods (progressive distillation, CD, LCM) copy the teacher’s ODE map. At 1 step this map is very hard to fit, and the student’s errors show up as blur. A second family drops the trajectory and asks only that the student’s samples look like real data or like the teacher’s distribution:

  • Adversarial Diffusion Distillation (ADD; Sauer et al., 2023) trains the student with a GAN loss plus a score-distillation loss from the teacher:
\[ \mathcal{L} = \mathcal{L}_{\text{adv}}^{G} + \lambda\, \mathcal{L}_{\text{distill}}, \qquad \lambda = 2.5 \tag{230.12} \]

The discriminator is a frozen DINOv2 ViT-S feature network with small trainable heads; the student uses 4 noise levels and can sample in 1 to 4 steps. ADD-XL (3.1B parameters, distilled from SDXL) beats LCM-XL at 4 steps with a single step in their human study, and beats SDXL at 50 steps with 4 steps in most comparisons. [Certain] The released SDXL-Turbo is an ADD-distilled SDXL that works at 512 × 512 with guidance_scale=0.0. [Certain]

  • Distribution Matching Distillation (DMD; Yin et al., 2023) makes a one-step generator \(G_\theta\) whose output distribution has low KL divergence to the teacher’s. The gradient needs only two score functions: the real one (the frozen teacher) and a “fake” one, a second diffusion model trained online on the generator’s outputs:
\[ \nabla_\theta \KL \approx \E_{\mathbf{z}, t}\Big[ w_t\, \big( s_{\text{fake}}(\mathbf{x}_t, t) - s_{\text{real}}(\mathbf{x}_t, t) \big)\, \frac{\partial G_\theta(\mathbf{z})}{\partial \theta} \Big], \quad \mathbf{x}_t = G_\theta(\mathbf{z}) + t\,\varepsilon \tag{230.13} \]

The difference of scores pushes each generated sample away from where the generator is too dense and toward where the real data is denser. DMD adds a regression loss on a set of teacher (noise, image) pairs to keep the large-scale structure. It reports FID 2.62 on ImageNet 64 × 64 and 11.49 on zero-shot COCO-30k, and 20 images per second with FP16 on modern hardware. [Certain]

  • Later variants. SDXL-Lightning (Lin et al., 2024) combines progressive and adversarial distillation for 1024 px SDXL and releases LoRA and full weights. [Certain] LADD (Sauer et al., 2024) moves the discriminator into latent space, using features of a pretrained latent diffusion model instead of DINOv2, and makes SD3-Turbo (from the 8B SD3) with 4 unguided steps. [Certain] FLUX.1 schnell, a 12B rectified-flow transformer under Apache-2.0, is “trained using latent adversarial diffusion distillation” and makes images in 1 to 4 steps; its example code uses 4 steps and guidance_scale=0.0. [Certain]

Straighter paths

The third route makes the path itself easier. If trajectories were straight lines, one Euler step would be exact. Rectified flow (session 229) “reflows” a trained model on its own (noise, image) pairs to straighten paths, then distills. InstaFlow (Liu et al., 2023) used this on Stable Diffusion: one step, FID 23.3 on MS COCO 2017-5k against 37.2 for progressive distillation, about 0.09 s per image, for about 199 A100 GPU-days of training. [Certain]

1. Better solvers (no training) DDIM, Heun, DPM-Solver++, UniPCSame network, smarter stepsGood at 10 to 25 stepsFails below about 8 steps 2. Copy the trajectory Progressive distillation, CD, LCMStudent matches the teacher’s ODE mapKeeps diversity; bounded by teacherBlurs at 1 step if capacity is short 3. Match the distribution ADD, LADD, DMD, LightningGAN or score-difference lossSharpest 1 to 4 step samplesRisk: less variety, harder training 4. Straighten the path Rectified flow, reflow, InstaFlowStraight paths: Euler is nearly exactOften combined with 2 or 3Reflow needs a big paired dataset
Figure 7. Four families of methods for fewer network calls. Real systems mix them: FLUX.1 schnell is a rectified-flow model (4) distilled adversarially (3); LCM is a trajectory method (2) that also bakes in guidance.

Quality vs steps vs latency

Every few-step method trades something. The LCM paper gives a clean comparison on Stable Diffusion with the same guidance scale (\(\omega = 8\)): FID (image quality and variety against real images, lower is better) and CLIP score (how well the image matches the prompt, higher is better) at 1, 2, 4 and 8 steps, for two training-free solvers, a guided progressive distillation baseline and LCM. Figure 8 plots their Tables 1 and 2; the evaluation uses 30K images from 10K prompts of LAION-Aesthetics. [Certain]

DDIMDPM-Solver++Guided-DistillLCM
Figure 8. Published numbers from Luo et al. (2023), Tables 1 (512 × 512, LAION-Aesthetics-6+) and 2 (768 × 768, LAION-Aesthetics-6.5+). The DPM-Solver row of the paper is almost identical to DPM-Solver++ and is left out.
How to read Figure 8

At 1 and 2 steps, the solvers are useless (FID 183 and 81 for DDIM at 512 px) and LCM is far ahead (35.4 and 13.3). At 4 steps LCM is still about 2 times better than DDIM (11.10 vs 22.38). At 8 steps the solvers almost catch up, and in CLIP score they pass LCM (29.29 for DDIM vs 28.84). At 768 px, LCM’s FID gets slightly worse from 4 to 8 steps (13.53 to 14.97). Distilled models are built for a few steps; more steps do not always help them.

How to choose

  • Quality per step is not quality. Distilled models give much better images at 1 to 4 steps, but the teacher at 25 to 50 steps is usually still the best, especially in fine detail, text rendering and variety. [Likely] Keep the teacher for “high quality” mode and use the student for previews and real-time modes.
  • Diversity. Trajectory methods keep the teacher’s map from noise to image, so different seeds give different images. Adversarial and mode-seeking objectives can reduce variety (DMD explicitly adds a regression loss to keep the structure of the teacher’s outputs). Measure diversity, not just FID or a preference score. [Likely]
  • Guidance is frozen in. LCM learns a range of \(\omega\); SDXL-Turbo and FLUX.1 schnell are used with guidance off. Negative prompts and strong guidance edits, which need the unconditional branch, may not work the same way. [Likely]
  • Steps you can still choose. Consistency models and LCMs let you pick 1, 2, 4 or 8 steps at sampling time. Progressive distillation gives a fixed grid per student (but you can keep the students from all rounds).
  • Latency is not NFE. With 1 to 4 calls, the VAE decoder, text encoder, memory transfers and Python overhead matter (Figure 1). Profile the full pipeline; CUDA graphs and compilation help here (session 127).
  • Video multiplies everything. One call of a video model processes all frames of a clip, so a 50-step video sample is very expensive, and few-step distillation is even more valuable there (session 231). Temporal consistency must survive the distillation: a student that is sharp per frame but flickers is a failure. [Likely]

Few-step distillation in PyTorch

A complete, deterministic example on the 1-D mixture. The teacher is the exact denoiser, so all remaining error comes from the student network and the method. Part 1 trains a consistency model by consistency distillation (Heun teacher, \(N = 18\), squared L2 distance, EMA target) and samples it with 1 and 2 steps. Part 2 runs four rounds of progressive distillation, 16 → 8 → 4 → 2 → 1 steps, with a small MLP student. It runs in about 15 seconds on a laptop CPU.

import math, torch, torch.nn as nn
torch.manual_seed(0)

# toy data: a 1-D mixture of three Gaussians (same as the Diffusion 2 session)
W  = torch.tensor([0.35, 0.45, 0.20])
MU = torch.tensor([-2.0, 0.5, 2.5])
SD = torch.tensor([0.3, 0.4, 0.2])
SMIN, SMAX, RHO = 0.002, 80.0, 7.0
SIG_D = 0.5                                       # sigma_data in c_skip, c_out (the CM default)

def sample_data(n):
    c = torch.multinomial(W, n, replacement=True)
    return MU[c] + SD[c] * torch.randn(n)

def D_teacher(x, s):
    """The exact denoiser E[x0 | x_sigma] of the mixture: a perfect teacher."""
    v = SD**2 + s[:, None]**2
    logw = torch.log(W) - 0.5 * torch.log(v) - 0.5 * (x[:, None] - MU)**2 / v
    r = torch.softmax(logw, dim=1)
    return (r * (MU + SD**2 / v * (x[:, None] - MU))).sum(1)

def karras(N):                                    # sigma grid, ascending: t_1 = eps ... t_N = T
    i = torch.arange(N, dtype=torch.float64)
    return ((SMIN**(1/RHO) + i/(N-1) * (SMAX**(1/RHO) - SMIN**(1/RHO)))**RHO).float()

def heun(x, a, b):                                # one Heun step of the PF-ODE, sigma a -> b
    d = (x - D_teacher(x, a)) / a
    xe = x + (b - a) * d
    d2 = (xe - D_teacher(xe, b)) / b
    return x + (b - a) * (d + d2) / 2

class F(nn.Module):                               # tiny network F_theta(x, sigma)
    def __init__(s, h=128):
        super().__init__()
        s.net = nn.Sequential(nn.Linear(2, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(),
                              nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 1))
    def forward(s, x, sig):
        c_in = 1 / torch.sqrt(sig**2 + SIG_D**2)
        return s.net(torch.stack([c_in * x, torch.log(sig) / 4], 1)).squeeze(1)

def f_cm(model, x, sig):                          # formula (230.6): boundary f(x, eps) = x built in
    c_skip = SIG_D**2 / ((sig - SMIN)**2 + SIG_D**2)
    c_out = SIG_D * (sig - SMIN) / torch.sqrt(SIG_D**2 + sig**2)
    return c_skip * x + c_out * model(x, sig)

# ---- consistency distillation, formula (230.7) ----
N, MU_EMA, STEPS, B = 18, 0.95, 4000, 512
t = karras(N)
model = F(); target = F(); target.load_state_dict(model.state_dict())
for p in target.parameters(): p.requires_grad_(False)
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for it in range(STEPS):
    x0 = sample_data(B)
    n = torch.randint(0, N - 1, (B,))             # pair (t_n, t_{n+1})
    a, b = t[n + 1], t[n]
    xa = x0 + a * torch.randn(B)                  # a point on the noisy marginal at t_{n+1}
    with torch.no_grad():
        xb = heun(xa, a, b)                       # one teacher ODE step down to t_n
        tgt = f_cm(target, xb, b)                 # f_{theta^-}(x_hat_{t_n}, t_n)
    loss = ((f_cm(model, xa, a) - tgt)**2).mean() # squared L2 distance, lambda = 1
    opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():                         # EMA target network
        for pt, pm in zip(target.parameters(), model.parameters()):
            pt.mul_(MU_EMA).add_(pm, alpha=1 - MU_EMA)

# ---- sampling and evaluation (1-D Wasserstein-1 against fresh data) ----
def w1(x, ref): return (x.sort().values - ref.sort().values).abs().mean().item()
g = torch.Generator().manual_seed(1)
M = 20000
ref = sample_data(M)
zT = SMAX * torch.randn(M, generator=g)
with torch.no_grad():
    one = f_cm(model, zT, torch.full((M,), SMAX))                     # 1 NFE
    tau = 1.0
    xt = one + math.sqrt(tau**2 - SMIN**2) * torch.randn(M, generator=g)
    two = f_cm(model, xt, torch.full((M,), tau))                      # 2 NFE, formula (230.9)
    x = zT.clone()                                                    # DDIM with 2 NFE, same noise
    for a_, b_ in [(SMAX, 1.0), (1.0, 0.0)]:
        d = D_teacher(x, torch.full((M,), a_))
        x = d + (b_ / a_) * (x - d)
print(f"CD 1 step  W1 = {w1(one, ref):.3f}")
print(f"CD 2 steps W1 = {w1(two, ref):.3f}")
print(f"DDIM 2 steps (perfect denoiser) W1 = {w1(x, ref):.3f}")
print(f"data vs data floor W1 = {w1(sample_data(M), ref):.3f}")

# ---- progressive distillation, formulas (230.3) and (230.4): 16 -> 8 -> 4 -> 2 -> 1 steps ----
def karras_desc(N):                               # sampling grid, descending, with sigma_N = 0
    return torch.cat([karras(N).flip(0), torch.zeros(1)])

def ddim(x, a, b, denoise):                       # formula (230.2) in sigma form
    d = denoise(x, a)
    return d + (b / a) * (x - d)

class Student(nn.Module):                         # D_eta(x; sigma) = c_skip x + c_out F(x, sigma)
    def __init__(s):
        super().__init__(); s.F = F()
    def forward(s, x, sig):
        c_skip = SIG_D**2 / (sig**2 + SIG_D**2)
        c_out = sig * SIG_D / torch.sqrt(sig**2 + SIG_D**2)
        return c_skip * x + c_out * s.F(x, sig)

def sample(denoise, grid, z):
    x = z.clone()
    for i in range(len(grid) - 1):
        x = ddim(x, torch.full_like(x, grid[i]), torch.full_like(x, grid[i + 1]), denoise)
    return x

teacher, grid = D_teacher, karras_desc(16)       # round 0: the exact denoiser with 16 DDIM steps
with torch.no_grad():
    print(f"teacher 16-step W1 = {w1(sample(teacher, grid, zT), ref):.3f}")
student = Student()                               # round 1 starts from scratch; later rounds start from the last student
for rnd in range(4):
    new_grid = grid[::2]                          # every other sigma: half the steps
    opt = torch.optim.Adam(student.parameters(), lr=1e-3)
    for it in range(1500):
        j = torch.randint(0, len(new_grid) - 1, (B,))
        a, b = new_grid[j], new_grid[j + 1]
        mid = grid[2 * j + 1]                     # the teacher's in-between sigma
        xa = sample_data(B) + a * torch.randn(B)
        with torch.no_grad():                     # two teacher DDIM steps a -> mid -> b
            xb = ddim(ddim(xa, a, mid, teacher), mid, b, teacher)
            ok = b > 0
            x_tilde = torch.where(ok, (a * xb - b * xa) / (a - b), xb)   # (230.3)
            w = torch.clamp(1 / a**2, min=1.0)    # truncated SNR weight, max(SNR, 1)
        loss = (w * (student(xa, a) - x_tilde)**2).mean()               # (230.4)
        opt.zero_grad(); loss.backward(); opt.step()
    frozen = Student(); frozen.load_state_dict(student.state_dict())
    for p in frozen.parameters(): p.requires_grad_(False)
    teacher, grid = frozen, new_grid              # the student becomes the next teacher
    with torch.no_grad():
        print(f"student {len(grid)-1:2d}-step W1 = {w1(sample(teacher, grid, zT), ref):.3f}"
              f"   DDIM {len(grid)-1:2d}-step, perfect denoiser: {w1(sample(D_teacher, grid, zT), ref):.3f}")

Output of this script (PyTorch 2, CPU):

CD 1 step  W1 = 0.065
CD 2 steps W1 = 0.040
DDIM 2 steps (perfect denoiser) W1 = 0.702
data vs data floor W1 = 0.011
teacher 16-step W1 = 0.169
student  8-step W1 = 0.170   DDIM  8-step, perfect denoiser: 0.344
student  4-step W1 = 0.170   DDIM  4-step, perfect denoiser: 0.609
student  2-step W1 = 0.167   DDIM  2-step, perfect denoiser: 0.684
student  1-step W1 = 0.164   DDIM  1-step, perfect denoiser: 1.403

Read it the same way as the figures. The consistency model gives \(W_1 = 0.065\) in one call and 0.040 in two, while two DDIM steps with the perfect denoiser give 0.702. The distilled students keep the 16-step teacher’s quality (0.169) all the way down to one step (0.164; the small changes are training noise), while plain DDIM on the same grids degrades to 1.403. The floor 0.011 is the error between two independent sets of 20,000 data samples, so differences smaller than that mean nothing.

Where this is used

SettingMethodWhat matters
Real-time text-to-image (typing-speed previews, live drawing-to-image)ADD (SDXL-Turbo), LCM, LCM-LoRA: 1 to 4 stepsTotal pipeline latency, including decoder (Figure 1)
Open-weight fast image modelsFLUX.1 schnell (LADD, 1 to 4 steps, Apache-2.0); SD3-Turbo (LADD, 4 steps); SDXL-Lightning (LoRA or full weights)Guidance off; quality close to the teacher at 4 steps
Speeding up a community of fine-tunesLCM-LoRA as a plug-in adapterOne adapter, many styles; check quality per style
Image editing, inpainting, colorization, super-resolutionConsistency models support these zero-shot, without task-specific training [Certain]; the known pixels are put back between steps [Likely]Few steps make interactive editing possible
Video generationConsistency and adversarial distillation of video diffusion models into few-step samplers [Likely]Per-call cost is huge; must keep frames consistent over time
Synthetic data for video tasks (action recognition, retrieval, key-moment detection)Few-step generators make large synthetic sets affordable [Speculation]Diversity loss of distilled models can bias the data
Robot policies and VLA models that sample actions with diffusionConsistency-style or distilled policies for fast control loops [Likely]Control rate (for example 10 to 50 Hz) sets the step budget
Generated images inside a VLM or an any-to-any modelA few-step decoder after the language model (session 232)Image latency adds to the text answer latency

Common mistakes

  • Expecting a solver to fix 1 to 4 steps. With a pretrained, undistilled model, 1 DDIM step returns the posterior mean: a blur, whatever the solver. Below about 8 steps you need a distilled or consistency model.
  • Using a distilled model with the teacher’s sampler settings. LCM needs its own scheduler; Turbo and schnell models expect guidance off. Running CFG on top of a guidance-distilled model applies guidance twice and over-saturates images. [Likely]
  • More steps on a few-step model. Distilled models are tuned for their step counts; going from 4 to 8 often brings little, and sometimes hurts (Figure 8, 768 px).
  • \(\varepsilon\)-prediction for a 1-step student. At the highest noise level the clean guess from \(\hat\varepsilon\) divides by a near-zero \(\alpha_t\). Use \(\mathbf{x}\)-, v- or EDM-style skip parameterizations (230.7).
  • Forgetting the boundary condition. Without \(c_{\text{skip}}(\epsilon) = 1\), \(c_{\text{out}}(\epsilon) = 0\), the model can satisfy self-consistency with a constant output (all points map to the same image). The boundary condition is what ties the chain to real data.
  • Backpropagating through the target. The target \(f_{\theta^-}(\hat{\mathbf{x}}_{t_n}, t_n)\) in (230.8) must be computed without gradients; otherwise the model can lower the loss by moving the target and training becomes unstable.
  • Judging only by FID or a preference score. Check diversity across seeds, prompt following, text rendering and, for video, temporal flicker. One-step models can look great on average and fail on the long tail.
  • Measuring only the denoiser loop. At 1 to 4 steps the decoder and text encoder can be half the time; report end-to-end latency.

Interview questions

Why is diffusion sampling slow, and what are the main families of fixes?basic
Each sample needs many sequential calls of a large network: \(N\) steps, times 2 with classifier-free guidance, plus fixed costs (formula 230.1). Fixes: (1) better ODE solvers (DDIM, DPM-Solver++, UniPC) that cut steps to about 10 to 25 with no retraining; (2) trajectory distillation (progressive distillation, consistency distillation, LCM) that trains a student to take big steps along the teacher’s ODE; (3) distribution matching (ADD, LADD, DMD) that trains a 1 to 4 step generator with adversarial or score-difference losses; (4) straighter paths (rectified flow) so that few Euler steps are accurate. Products often combine them, and also distill guidance into the model to remove the second call per step.
Why does one DDIM step from pure noise give a blurry image, even with a perfect network?core
A one-step DDIM jump to \(\sigma = 0\) outputs the denoiser \(D(\mathbf{x}_T; T) = \E[\mathbf{x}_0 \mid \mathbf{x}_T]\). This is a posterior mean. At high noise, many very different images are consistent with \(\mathbf{x}_T\), so their average is a smooth, grey blur. The blur is not an error of the network; it is what minimizing squared error gives. Few-step models must learn a different function: the end point of one ODE trajectory (consistency function) or the result of many teacher steps (distillation target), which is a single sharp sample. On the 1-D toy, one DDIM step with the exact denoiser has \(W_1 = 1.41\); a one-step consistency model trained with a Heun teacher and \(N = 18\) has 0.040.
Explain progressive distillation. What is the student’s target?core
Start with a teacher and an \(N\)-step DDIM sampler. Initialize a student as a copy. For a noisy training input \(\mathbf{x}_a\), run two teacher DDIM steps to \(\mathbf{x}_b''\). The target is the clean guess that makes one student DDIM step from \(a\) land at \(\mathbf{x}_b''\): \(\tilde{\mathbf{x}} = (a\mathbf{x}_b'' - b\mathbf{x}_a)/(a - b)\) in the \(\sigma\) form (formula 230.3). Train with weighted squared error (truncated SNR or SNR+1 weighting). The student now samples in \(N/2\) steps. It becomes the next teacher; repeat \(\log_2\) times. On CIFAR-10: FID 3.00 at 4 steps and 9.12 at 1 step, starting from 8192 steps, with about 50k updates per round.
Why did progressive distillation need new parameterizations such as v-prediction?advanced
With \(\varepsilon\)-prediction, the clean estimate is \(\hat{\mathbf{x}} = (\mathbf{z}_t - \sigma_t\hat\varepsilon)/\alpha_t\). As the student’s steps get larger, it must work at the noisiest levels, where \(\alpha_t \to 0\) and SNR \(\to 0\). There, small errors in \(\hat\varepsilon\) are divided by a tiny \(\alpha_t\) and explode. Predicting \(\mathbf{x}\) directly, both \(\mathbf{x}\) and \(\varepsilon\), or \(\mathbf{v} = \alpha_t\varepsilon - \sigma_t\mathbf{x}\) (with \(\hat{\mathbf{x}} = \alpha_t\mathbf{z}_t - \sigma_t\hat{\mathbf{v}}\)) stays well behaved at every noise level. The loss weighting also changes: plain SNR weighting gives zero weight at zero SNR, so they use \(\max(\text{SNR}, 1)\) or \(\text{SNR} + 1\).
What is a consistency model? Define self-consistency and the boundary condition.basic
A consistency model learns \(f(\mathbf{x}_t, t)\), which maps any point of a probability-flow ODE trajectory, at any noise level \(t\), to the trajectory’s end point at \(t = \epsilon\). Self-consistency: points on the same trajectory give the same output, \(f(\mathbf{x}_t, t) = f(\mathbf{x}_{t'}, t')\). Boundary condition: \(f(\mathbf{x}, \epsilon) = \mathbf{x}\). It is built into the architecture with \(f_\theta = c_{\text{skip}}(t)\mathbf{x} + c_{\text{out}}(t)F_\theta\), where \(c_{\text{skip}}(\epsilon) = 1\), \(c_{\text{out}}(\epsilon) = 0\). Sampling is one call \(f_\theta(\mathbf{x}_T, T)\), and more calls can be added by re-noising (formula 230.10).
Consistency distillation vs consistency training: what is the difference, and what does each cost?core
CD uses a pretrained diffusion model: from \(\mathbf{x}_{t_{n+1}}\) it takes one teacher ODE step (Euler: 1 call, Heun: 2 calls) to get \(\hat{\mathbf{x}}_{t_n}\) on the same trajectory, and matches \(f_\theta(\mathbf{x}_{t_{n+1}})\) to \(f_{\theta^-}(\hat{\mathbf{x}}_{t_n})\). Its quality is bounded by the teacher and the solver error. CT needs no teacher: it uses the same noise \(\mathbf{z}\) on the same image at both levels, which is a one-sample estimate of the trajectory step (unbiased score estimate). CT is cheaper per step but noisier and was worse at first (CIFAR-10 one-step FID 8.70 vs 3.55); improved CT (iCT) reached 2.51, and continuous-time sCM reached two-step FID 1.88 on ImageNet 512 × 512.
In consistency distillation, why use a target network \(\theta^-\) with stop-gradient? What stops the model from outputting a constant?advanced
Self-consistency alone has a trivial solution: a constant function agrees with itself everywhere. The boundary condition \(f(\mathbf{x}, \epsilon) = \mathbf{x}\), built into the architecture, rules it out: at \(t_1 = \epsilon\) the target is the input itself, and the bootstrapping passes this correct anchor up to higher noise levels. The stop-gradient target (an EMA copy in the original paper) makes the problem a regression to a slowly moving target, like the target network in deep Q-learning; without it, the model could reduce the loss by moving both sides, which is unstable. iCT later found that \(\mu = 0\) for the target (no EMA, but still stop-gradient) is better in consistency training.
Why can 2-step consistency sampling beat 1 step, if the model already predicts the end point?core
The learned \(f_\theta\) is not perfect, and its error is largest at the highest noise level, where the map from noise to image is most complex. Two-step sampling takes the first sample, adds fresh noise to a middle level \(\tau\), and applies \(f_\theta\) again from there, where the model is more accurate. It can repair details while keeping the overall structure. On CIFAR-10 CD goes from FID 3.55 to 2.93; on the 1-D toy with Heun and \(N = 18\), \(W_1\) goes from 0.040 to 0.024 at \(\tau \approx 1.09\). The choice of \(\tau\) matters; the paper searches it greedily.
How does LCM deal with classifier-free guidance, and what is LCM-LoRA?core
LCM distills the guided ODE: the teacher step uses \(\tilde\varepsilon = (1 + \omega)\varepsilon(\mathbf{z}, \mathbf{c}) - \omega\varepsilon(\mathbf{z}, \varnothing)\), and \(\omega\) (drawn from \([2, 14]\)) is an input to the student. So at sampling time one call per step gives a guided result: no second, unconditional call. It also skips steps (consistency between \(t_{n+k}\) and \(t_n\) with \(k = 20\)) to shorten the bootstrapping chain. LCM-LoRA trains this as a LoRA adapter; the authors report it can be plugged into other fine-tunes of the same base model without retraining, which makes a whole family of models fast at once.
Compare trajectory distillation (CD, LCM) with adversarial or distribution-matching distillation (ADD, DMD).advanced
Trajectory methods regress onto the teacher’s ODE map. They keep the teacher’s noise-to-image correspondence, so diversity is preserved, training is stable, and quality is bounded by the teacher plus the student’s fitting error; at 1 step the map is very hard and results blur. ADD adds a GAN loss (frozen DINOv2 feature discriminator) plus score distillation (\(\lambda = 2.5\)); DMD minimizes an approximate KL using the difference of a real and a fake score (formula 230.13) plus a regression loss. These only require the samples to look like the data, so 1-step results are much sharper (ADD-XL at 1 step beats LCM-XL at 4 steps in their study), but they can lose variety (mode seeking), need a discriminator or an online fake-score model, and are harder to train. Production systems often combine both (SDXL-Lightning: progressive + adversarial).
Your text-to-image service takes about 5 s per image (50 steps with CFG). Product wants under 300 ms. Walk through your plan.core
First profile end to end (formula 230.1): with 50 ms per call and 100 ms fixed, 100 calls give 5.1 s. Options in order: (1) a 20-step multistep solver halves the calls, not enough; (2) distill guidance in (one call per step) and distill steps: LCM-LoRA or a consistency/adversarial student at 4 steps gives 4 calls, 300 ms at these numbers; 1 to 2 steps gives 150 to 200 ms. (3) Now the fixed part dominates: use a small or distilled VAE decoder, cache text embeddings, compile and use CUDA graphs, and batch. (4) Evaluate quality: FID or preference scores on your prompts, diversity across seeds, text rendering; keep the teacher as a “high quality” option. (5) Consider lower resolution plus an upscaler. Report latency at p50 and p99 under real load.
What changes when you distill a video diffusion model instead of an image model?advanced
Each call processes all frames (a space-time latent), so per-call cost and memory are much higher and the payoff of fewer calls is larger. But the student must also learn temporal coherence: per-frame sharpness can come with flicker or identity drift if the loss does not see the time dimension. Distribution-matching losses need a discriminator or fake-score model that looks at clips, not frames. Evaluation must include temporal metrics (FVD, human preference for motion and consistency) as in session 231. Data for distillation is also costlier: teacher samples for video are expensive to generate, which favours consistency methods that only need one teacher step per training pair. [Likely]

Exercises

  1. A progressive distillation run starts from a 1024-step teacher. How many rounds to reach 4 steps? To reach 1 step?
  2. Two teacher DDIM steps take \(\mathbf{x}_a = 0.8\) at \(a = 1\) to \(\mathbf{x}_b'' = 0.3\) at \(b = 0.5\). Compute the distillation target \(\tilde{\mathbf{x}}\) and check it.
  3. Compute \(c_{\text{skip}}\) and \(c_{\text{out}}\) of (230.7) with \(\sigma_{\text{data}} = 0.5\) and \(\epsilon = 0.002\) at \(t = \epsilon\) and \(t = 80\). What do they tell you?
  4. A model runs 30 steps with CFG, 40 ms per call and 80 ms fixed time. What is the latency? What is it for a 4-step guidance-distilled student, and what is the speed-up?
  5. In Figure 6, Heun at \(N = 3\) is much worse than Euler at \(N = 3\), but much better at \(N = 18\). Explain both.
  6. Show that in 1-D, \(f(x, \sigma) = Q_0(F_\sigma(x))\) satisfies self-consistency and the boundary condition.
Answerscheck
1. \(\log_2(1024/4) = 8\) rounds to 4 steps; \(\log_2 1024 = 10\) rounds to 1 step. 2. \(\tilde{\mathbf{x}} = (1 \times 0.3 - 0.5 \times 0.8)/(1 - 0.5) = -0.1/0.5 = -0.2\). Check with (230.2): \(-0.2 + 0.5 \times (0.8 - (-0.2)) = -0.2 + 0.5 = 0.3\). 3. At \(t = \epsilon\): \(c_{\text{skip}} = 0.25/(0 + 0.25) = 1\), \(c_{\text{out}} = 0\): the output is the input (boundary condition) for any network. At \(t = 80\): \(c_{\text{skip}} = 0.25/(79.998^2 + 0.25) \approx 0.00004\), \(c_{\text{out}} = 0.5 \times 79.998/\sqrt{0.25 + 6400} \approx 0.49998\): the output is almost only the network, scaled to the data’s size. 4. \(30 \times 2 \times 40 + 80 = 2480\) ms; \(4 \times 1 \times 40 + 80 = 240\) ms; speed-up \(2480/240 \approx 10.3\times\) (NFE ratio 15, the fixed 80 ms eats the rest). 5. With \(N = 3\) the middle level is about 2.5, so the teacher step goes from 80 to 2.5 (and then 2.5 to 0.002). Heun evaluates the slope at the end of a huge Euler step, at a badly wrong point, and averages it in; on such long, curved steps this second slope can be worse than none (\(W_1\) 5.99 vs 0.78). With \(N = 18\), steps are short, Heun’s error is second order (about \(1/N^2\)) and it wins clearly (0.040 vs 0.153). 6. On one trajectory, the ODE preserves quantiles: if \(x_\sigma\) is at quantile \(u\) of \(p_\sigma\), then \(F_\sigma(x_\sigma) = u\) for every \(\sigma\) along the path, so \(f = Q_0(u)\) is the same at every level (self-consistency). At \(\sigma \to 0\), \(F_0\) and \(Q_0\) are inverses, so \(f(x, 0) = x\) (boundary condition; at \(\epsilon = 0.002\) it is true up to the tiny noise).

Key takeaways

  • Latency is \(T_{\text{fixed}} + N c\, t_{\text{net}}\) (230.1). Few-step methods cut \(N\) and remove the CFG factor \(c\); then fixed costs dominate.
  • One DDIM step (230.2) returns the posterior mean, a blur, even with a perfect network. Few-step generation needs a network trained for big steps.
  • Progressive distillation (230.3, 230.4) halves the steps per round by matching two teacher steps with one; quality stays near the teacher if the student has the capacity (Figure 2). Use x-, v- or skip parameterizations, not \(\varepsilon\).
  • Consistency models learn \(f(\mathbf{x}_t, t)\) = the end of the ODE path (230.5), with the boundary condition built in (230.7); CD bootstraps with one teacher step and a stop-gradient target (230.8), CT uses shared noise instead of a teacher (230.9). Multistep sampling (230.10) trades calls for quality.
  • Products: LCM and LCM-LoRA (guidance in the ODE, 230.11), ADD and LADD (adversarial, 230.12), DMD (score difference, 230.13), and straight rectified-flow paths. FLUX.1 schnell and SDXL-Turbo run in 1 to 4 steps with guidance off.
  • Judge few-step models on quality, diversity, prompt following and end-to-end latency, and for video on temporal coherence.

Key formulas

\[ \tilde{\mathbf{x}} = \frac{a\,\mathbf{x}_b'' - b\,\mathbf{x}_a}{a - b}, \qquad f_\theta = c_{\text{skip}}(t)\,\mathbf{x} + c_{\text{out}}(t)\,F_\theta(\mathbf{x}, t) \]
\[ \mathcal{L}_{\text{CD}} = \E\big[ d\big(f_\theta(\mathbf{x}_{t_{n+1}}, t_{n+1}),\ f_{\theta^-}(\hat{\mathbf{x}}_{t_n}, t_n)\big) \big], \qquad \hat{\mathbf{x}}_{\tau} = \mathbf{x} + \sqrt{\tau^2 - \epsilon^2}\,\mathbf{z} \]

References for this session

  1. Salimans & Ho (2022). Progressive Distillation for Fast Sampling of Diffusion Models
  2. Song et al. (2023). Consistency Models
  3. Song et al. (2020). Denoising Diffusion Implicit Models
  4. Ho & Salimans (2022). Classifier-Free Diffusion Guidance
  5. Liu et al. (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
  6. Karras, Aittala, Aila & Laine (2022). Elucidating the Design Space of Diffusion-Based Generative Models
  7. Song & Dhariwal (2023). Improved Techniques for Training Consistency Models
  8. Lu & Song (2024). Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models
  9. Luo, Tan, Huang, Li & Zhao (2023). Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
  10. Luo, Tan, Patil, Gu, von Platen, Passos, Huang, Li & Zhao (2023). LCM-LoRA: A Universal Stable-Diffusion Acceleration Module
  11. Meng, Rombach, Gao, Kingma, Ermon, Ho & Salimans (2023). On Distillation of Guided Diffusion Models
  12. Sauer, Lorenz, Blattmann & Rombach (2023). Adversarial Diffusion Distillation
  13. Sauer, Boesel, Dockhorn, Blattmann, Esser & Rombach (2024). Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation
  14. Yin, Gharbi, Zhang, Shechtman, Durand, Freeman & Park (2024). One-step Diffusion with Distribution Matching Distillation
  15. Lin, Wang & Yang (2024). SDXL-Lightning: Progressive Adversarial Diffusion Distillation
  16. Liu, Zhang, Ma, Peng & Liu (2023). InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation
  17. Black Forest Labs (2024). FLUX.1 [schnell] model card