Few-step generation: consistency models and distillation
A diffusion model needs 20 to 100 network calls per image, and that is too slow for many products. This session shows why a better solver cannot go below a handful of steps, and how to train a network that takes big steps instead: progressive distillation halves the steps round after round, and consistency models jump from any noise level straight to the clean image. We then follow these ideas into real systems (LCM, LCM-LoRA, SDXL-Turbo, FLUX.1 schnell) and weigh quality against steps and latency.
What you will learn
- Where the time of a diffusion sample goes, and why the speed-up in latency is smaller than the drop in network calls
- Why one DDIM step gives a blurry posterior mean even with a perfect network
- Progressive distillation: the two-steps-into-one target, the loss weighting, the v-parameterization and the cost in rounds
- Consistency models: the consistency function, self-consistency, the built-in boundary condition, and multistep sampling
- Consistency distillation vs consistency training, why the target network uses stop-gradient, and how iCT and sCM improved them
- How LCM, LCM-LoRA, ADD, LADD, DMD and rectified flow turn these ideas into 1 to 4 step products
- How to choose between quality, diversity, step count and end-to-end latency, for images and for video
Why this matters
A diffusion model makes an image by calling a large network again and again: 20 to 50 times with a good solver, and twice per step when classifier-free guidance is on. For a chat-style product, a phone, or a video model that must make every frame of a clip, that is too slow and too expensive. This session is about the methods that bring the count down to 1 to 4 network calls.
In session 226 you saw that sampling is solving an ordinary differential equation (ODE) from noise to data, and that better solvers get good results in about 10 to 25 steps. Below about 8 steps, solvers fail: the path from noise to data is curved, and a few big straight steps miss it. The fix is not a better solver; it is a new network that is trained to take big steps. There are two classic ways to train it, and they are the core of this session:
- Progressive distillation (Salimans and Ho, 2022): teach a student to do in one step what the teacher does in two. Repeat, halving the step count each round.
- Consistency models (Song et al., 2023): teach a network to jump from any point on the noise-to-data path straight to the end of that path.
Then we look at how these ideas scaled to text-to-image products (Latent Consistency Models, LCM-LoRA, SDXL-Turbo, FLUX.1 schnell), at the adversarial and distribution-matching methods that won the 1-step race, and at the honest trade-off between quality, step count, diversity and latency.
The link to LLMs is direct. In session 59 a small student copies a big teacher’s outputs. Here the student copies the teacher’s sampler: many small steps become a few large ones. It is the same economics as session 128: the expensive model is called fewer times per output.
Where the time goes
Let \(N\) be the number of sampling steps. Each step calls the denoising network (a U-Net or a diffusion transformer) once, or twice with classifier-free guidance (CFG), because CFG runs the model with and without the prompt (session 227). Around the loop there is fixed work: the text encoder runs once, and for a latent model the VAE decoder turns the final latent into pixels once. So:
- Read the symbols
- \(N\)
- sampling steps
- \(c\)
- network calls per step: 2 with CFG, 1 without (or when guidance is built into the model)
- \(t_{\text{net}}\)
- time of one network call; for a large transformer at 1024 × 1024 this is often tens of milliseconds or more on one GPU [Likely]
- \(T_{\text{fixed}}\)
- text encoding, VAE decoding, safety checks and other work done once per image
- NFE
- number of function evaluations, the standard cost unit for samplers
With 50 steps and CFG, NFE = 100. A 4-step model with guidance built in has NFE = 4: 25 times fewer calls. But the speed-up in wall-clock time is smaller, because \(T_{\text{fixed}}\) does not shrink. This is Amdahl’s law: once the loop is short, the decoder and text encoder dominate. Figure 1 lets you set your own numbers. The defaults (50 ms per call, 100 ms fixed) are only an example, not a measurement; measure your own model as in session 123.
How to read Figure 1
At the defaults the teacher needs 100 calls and 5,100 ms; a 4-step model without CFG needs 300 ms (17 times faster, not 25, because of the 100 ms fixed part); a 1-step model needs 150 ms, two thirds of which is fixed time. Move the fixed time to 0 to see the pure NFE ratio. This is why few-step products also use a small or distilled VAE decoder and cache the text embeddings. [Likely]
Why a better solver is not enough
Recall the DDIM step from session 226 in the \(\sigma\) form, where a noisy sample is \(\mathbf{x}_\sigma = \mathbf{x}_0 + \sigma \varepsilon\). The denoiser \(D(\mathbf{x}; \sigma)\) is the network’s best guess of the clean sample. One step from noise level \(a\) to a lower level \(b\) is:
In words: jump to the clean guess, then add back a fraction \(b/a\) of the noise that was removed. If you go to \(b = 0\) in one step, the output is just \(D(\mathbf{x}_a; a)\). But the best possible denoiser is the average of all clean images that could have produced \(\mathbf{x}_a\): \(D(\mathbf{x}; \sigma) = \E[\mathbf{x}_0 \mid \mathbf{x}_\sigma = \mathbf{x}]\). At high noise, almost every image could have produced \(\mathbf{x}_a\), so the average is a grey blur. Even a perfect network gives a blurry image in one DDIM step. It is not a training problem; it is what the denoiser is defined to be.
On the 1-D toy of session 226 (a mixture of three Gaussians with means −2, 0.5 and 2.5, and its exact denoiser) we measure sample quality with the 1-D Wasserstein-1 distance \(W_1\): the average distance between the sorted samples and the true data quantiles (0 is perfect). DDIM with the perfect denoiser and the EDM noise schedule (\(\sigma_{\max} = 80\), \(\sigma_{\min} = 0.002\), \(\rho = 7\)) gives \(W_1 = 1.41\) at 1 step, 0.92 at 4 steps, 0.38 at 8, 0.18 at 16 and 0.042 at 64. Each halving of steps roughly doubles the error, until the samples collapse to the middle. Every method below beats this curve by training the network for big steps.
Progressive distillation: halve the steps again and again
Start with a trained teacher and a deterministic sampler with \(N\) steps, such as DDIM. Make a student network, initialized as a copy of the teacher. Train the student so that one of its steps lands where two teacher steps land. Now the student samples in \(N/2\) steps with nearly the teacher’s quality. Make the student the new teacher and repeat. After \(k\) rounds, the sampler has \(N / 2^k\) steps.
Why is one round easy to learn? Because two adjacent small steps are almost one straight step: the student has to learn only a small correction. Learning a 1000-step to 1-step map in one go is much harder; the paper’s point is that 2 to 1 is always a gentle task.
The training target
The student is a denoiser \(D_\eta\), used inside the same DDIM update (230.2). Suppose the teacher goes from \(\mathbf{x}_a\) to an intermediate level and then to \(\mathbf{x}_b''\), in two DDIM steps. Which clean guess \(\tilde{\mathbf{x}}\) would make one DDIM step from \(a\) land exactly at \(\mathbf{x}_b''\)? Put \(\tilde{\mathbf{x}}\) in place of \(D\) in (230.2) and solve \(\mathbf{x}_b'' = \tilde{\mathbf{x}} + \tfrac{b}{a}(\mathbf{x}_a - \tilde{\mathbf{x}})\):
The second form is the one in the paper, written for the variance-preserving process \(\mathbf{z}_t = \alpha_t \mathbf{x} + \sigma_t \varepsilon\); with \(\alpha = 1\) it is the first form. [Certain] The student is trained with a weighted squared error to this target:
- Read the symbols
- \(\mathbf{x}_a\)
- a training image with noise of level \(a\) added: \(\mathbf{x}_0 + a\,\varepsilon\); \(a\) is one of the student’s noise levels
- \(\mathbf{x}_b''\)
- where two DDIM steps of the (frozen) teacher end, starting from \(\mathbf{x}_a\)
- \(\tilde{\mathbf{x}}\)
- the target: the clean guess that makes one student step match the two teacher steps; it is a regression target, computed without gradients
- SNR
- signal-to-noise ratio, \(\alpha^2/\sigma^2\) (here \(1/a^2\)); “truncated SNR” weighting \(\max(\text{SNR}, 1)\) is one of two weightings proposed in the paper (the other is \(\text{SNR} + 1\)); plain SNR weighting gives zero weight at the noisiest level, which is exactly where the 1-step student must work
On the 1-D toy, take \(\mathbf{x}_a = 1.2\) at \(a = 2\). The teacher uses two DDIM steps, \(2 \to 1 \to 0.5\).
Step 1: \(D(1.2; 2) = 0.635\) (the posterior gives the three modes weights 0.146, 0.615 and 0.239, so the guess is an average). Then \(\mathbf{x}_1 = 0.635 + \tfrac12 (1.2 - 0.635) = 0.918\).
Step 2: \(D(0.918; 1) = 0.767\), then \(\mathbf{x}_b'' = 0.767 + \tfrac12 (0.918 - 0.767) = 0.843\).
Target (230.3): \(\tilde{\mathbf{x}} = (2 \times 0.843 - 0.5 \times 1.2) / 1.5 = 0.723\). Check: one DDIM step with this guess gives \(0.723 + \tfrac14 (1.2 - 0.723) = 0.843\), the two-step result. One plain teacher step \(2 \to 0.5\) would land at 0.776 instead.
Compare the three “clean guesses”: the denoiser says 0.635, the target says 0.723, and the true end of the ODE path through \((1.2, \sigma = 2)\) is 0.716. The target is not the average clean image any more; it already points toward the end of the path. Each round pushes this further.
- input: trained teacher \(D_\theta\); its sampler with \(N\) steps on noise levels \(\sigma_0 \gt \sigma_1 \gt \dots \gt \sigma_N = 0\)
- while \(N \gt\) target steps:
- student \(D_\eta \leftarrow\) copy of \(D_\theta\) // warm start: the student is already a good denoiser
- repeat (for example 50k updates):
- pick a data sample \(\mathbf{x}_0\), a student level \(a = \sigma_{2j}\), set \(b = \sigma_{2j+2}\); \(\mathbf{x}_a \leftarrow \mathbf{x}_0 + a\,\varepsilon\)
- \(\mathbf{x}_b'' \leftarrow\) two teacher DDIM steps \(a \to \sigma_{2j+1} \to b\) // 2 teacher calls, no gradients
- \(\tilde{\mathbf{x}} \leftarrow (a\,\mathbf{x}_b'' - b\,\mathbf{x}_a)/(a - b)\) (if \(b = 0\): \(\tilde{\mathbf{x}} \leftarrow \mathbf{x}_b''\)) // (230.3)
- gradient step on \(w(a)\,\lVert D_\eta(\mathbf{x}_a; a) - \tilde{\mathbf{x}} \rVert^2\) // (230.4)
- \(D_\theta \leftarrow D_\eta\); keep every other noise level; \(N \leftarrow N/2\) // the student becomes the teacher
- return \(D_\theta\) and its \(N\)-step grid
Figure 2 runs this on the 1-D toy. The teacher is 64 DDIM steps with the exact denoiser. Each student step is a lookup table with \(K\) points (placed at quantiles of the noisy data at that level) and straight lines between them; \(K\) plays the role of network capacity. The student is fitted to the exact two-step target at its points.
How to read Figure 2
Step through the rounds with \(K = 64\). The student’s error stays almost flat: 0.0416 (teacher), 0.0445 at 32 steps, 0.0474 at 1 step. Plain DDIM on the same noise levels grows from 0.084 to 1.41: at 1 step the distilled student is about 30 times better. Now pick \(K = 8\): the first round already loses quality (0.114) and every further round adds a little (0.143 at 1 step). With \(K = 256\) the student is essentially perfect (0.041 at every round). Lesson: distillation preserves the teacher’s quality only as far as the student can represent the sharper map; errors from each round are inherited by the next.
Details that matter in practice
- Parameterization. With \(\varepsilon\)-prediction, the clean guess is \(\hat{\mathbf{x}}_0 = (\mathbf{z}_t - \sigma_t \hat\varepsilon)/\alpha_t\). At the noisiest level \(\alpha_t \to 0\), so tiny errors in \(\hat\varepsilon\) are blown up. A 1-step student lives exactly there. The paper therefore predicts \(\mathbf{x}\) directly, or both \(\mathbf{x}\) and \(\varepsilon\), or v-prediction, \(\mathbf{v} = \alpha_t \varepsilon - \sigma_t \mathbf{x}\), with \(\hat{\mathbf{x}} = \alpha_t \mathbf{z}_t - \sigma_t \hat{\mathbf{v}}\). [Certain] v-prediction later became common in many diffusion models. [Likely]
- Cost. Each round is 50k updates in the paper (100k for the rounds that reach 2 and 1 steps). CIFAR-10 starts from an 8192-step teacher; the larger datasets start from 1024 steps. The authors argue the whole procedure costs no more than training the original model. [Certain] From 8192 to 4 steps is \(\log_2(8192/4) = 11\) rounds.
- Results. On CIFAR-10 the distilled models reach FID 9.12 at 1 step, 4.51 at 2, 3.00 at 4 and 2.57 at 8 steps. [Certain] (FID, the Fréchet Inception Distance, compares feature statistics of generated and real images; lower is better.)
- Guided models. Text-to-image models use CFG, so a plain teacher needs 2 calls per step. Meng et al. (2023) first distill the guided output (with the guidance scale as an input) into one network, then distill progressively. They report 1 to 4 denoising steps for latent models such as Stable Diffusion and at least 10 times faster inference than existing methods on ImageNet 256 × 256 and LAION. [Certain]
Consistency models: jump from any noise level to the clean image
Progressive distillation needs \(\log_2 N\) rounds and gives one fixed grid of steps. Consistency models (Song, Dhariwal, Chen and Sutskever, 2023) learn the end point of the ODE directly, in one training run.
The probability-flow ODE of session 226 is deterministic: every starting noise \(\mathbf{x}_T\) traces one path (a trajectory) down to one clean sample. So there is a function that takes any point on a trajectory, at any noise level, and returns that trajectory’s end point. That is the consistency function:
- Read the symbols
- \(t\)
- the noise level (the paper writes \(t\) for what we called \(\sigma\); in the EDM setup they are equal)
- \(\epsilon, T\)
- the smallest and largest noise levels; the paper uses \(\epsilon = 0.002\) and \(T = 80\) [Certain]
- \(f(\mathbf{x}_t, t) = f(\mathbf{x}_{t'}, t')\)
- self-consistency: two points on the same trajectory must give the same answer
- \(f(\mathbf{x}, \epsilon) = \mathbf{x}\)
- the boundary condition: at (almost) zero noise, the answer is the input itself
A network \(f_\theta\) that learns this function is a consistency model. Sampling is one call: draw \(\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, T^2 I)\) and output \(f_\theta(\mathbf{x}_T, T)\).
The difference from the denoiser is the whole point. The denoiser \(D(\mathbf{x}; \sigma)\) returns the average of all clean images consistent with \(\mathbf{x}\). The consistency function returns one specific clean image: the one the ODE would reach. At low noise the two agree. At high noise the average is a blur, but the trajectory end is a sharp, real-looking sample. Figure 3 shows both on the exact trajectories of the 1-D toy. In 1-D the probability-flow ODE never lets two paths cross, so it keeps the order of points: the path through the \(u\)-quantile of the noisy distribution ends at the \(u\)-quantile of the data. That gives the exact consistency function for the toy:
where \(F_\sigma\) is the cumulative distribution function (CDF) of the noisy data at level \(\sigma\) and \(Q_0\) is the quantile function (inverse CDF) of the clean data.
- pick one trajectory: the \(u\)-quantile path; walk \(\sigma\) from \(T\) down to \(\epsilon\)
- \(x_\sigma \leftarrow Q_\sigma(u)\) // the point of this path at level \(\sigma\)
- \(f \leftarrow Q_0(F_\sigma(x_\sigma))\) // (230.6): the end of the path
- \(D \leftarrow \E[x_0 \mid x_\sigma]\) // the denoiser: an average
- \(F \leftarrow (f - c_{\text{skip}}(\sigma)\, x_\sigma) / c_{\text{out}}(\sigma)\) // what the network must output, (230.7)
How to read Figure 3
Press Play (or Step). The animation visits nine noise levels on the 67% trajectory, from \(\sigma = 80\) down to \(\sigma = 0.002\). At each level it runs the four lines of the box: it finds the point \(x_\sigma\), computes \(f\) with (230.6), computes the denoiser \(D\), and computes what the network \(F\) must output. The table on the right fills up: the \(f\) column is 0.723 in every row (self-consistency), while \(D\) is 0.041 at \(\sigma = 80\), 0.645 at \(\sigma = 2\), 0.824 at \(\sigma = 1\) and 0.723 only at the end. The \(F\) column falls from 1.443 to 0.102 as \(c_{\text{skip}}\) grows from 0 to 1. You can also use the sliders at any time. Keep the trajectory fixed and move \(\sigma\): the point slides along the path and the round dot at the left never moves. The diamond (the denoiser) moves: 0.648 at \(\sigma = 0.5\), 0.645 at \(\sigma = 2\), 0.151 at \(\sigma = 10\), sliding toward the overall data mean (0.025) as the noise grows. Move \(\sigma\) to the left end: \(c_{\text{skip}} = 1\), \(c_{\text{out}} = 0\) and the output is the input (the boundary condition). Move a trajectory near 50%: it sits between two modes at high noise and only decides late (below \(\sigma \approx 1\)) where to go.
Building the boundary condition into the network
The boundary condition \(f(\mathbf{x}, \epsilon) = \mathbf{x}\) is not learned; it is built in with a skip connection, in the style of EDM (Karras et al., 2022):
- Read the symbols
- \(F_\theta\)
- the actual neural network (a U-Net or a transformer), free of constraints
- \(\sigma_{\text{data}}\)
- the standard deviation of the data; 0.5 for images scaled to \([-1, 1]\) in the paper [Certain]
- \(c_{\text{skip}}, c_{\text{out}}\)
- mixing weights: at \(t = \epsilon\) they are exactly 1 and 0, so \(f_\theta(\mathbf{x}, \epsilon) = \mathbf{x}\) for any weights \(\theta\); at \(t = 2\) they are 0.0589 and 0.4846; at \(t = 80\) they are about 0.00004 and 0.5, so the output comes almost only from the network
Consistency distillation (CD)
How do we train \(f_\theta\) without knowing the trajectories? Use a pretrained diffusion model as a teacher, and enforce self-consistency on neighbouring points only. Split \([\epsilon, T]\) into \(N\) levels \(t_1 = \epsilon \lt t_2 \lt \dots \lt t_N = T\). Take a training image, add noise of level \(t_{n+1}\), run one teacher ODE step down to \(t_n\), and ask the model to give the same answer at both points:
- Read the symbols
- \(\mathbf{x}_{t_{n+1}}\)
- \(\mathbf{x}_0 + t_{n+1} \mathbf{z}\) for a training image \(\mathbf{x}_0\) and \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\)
- \(\Phi\)
- the teacher ODE solver’s update direction; for Euler it is \((\mathbf{x} - D(\mathbf{x}; t))/t\), the DDIM step (230.2); the paper’s best CIFAR-10 setting used Heun (2 teacher calls) with \(N = 18\) [Certain]
- \(d\)
- a distance: squared L2, L1, or LPIPS (a learned perceptual distance); LPIPS was best on CIFAR-10 in the paper [Certain]
- \(\lambda(t_n)\)
- a weight; \(\lambda \equiv 1\) worked well [Certain]
- \(\theta^-\)
- the target network: a slowly moving copy, \(\theta^- \leftarrow \text{stopgrad}(\mu\, \theta^- + (1 - \mu)\, \theta)\); no gradient flows through it
Why does this work? At \(n = 1\) the target is \(f(\hat{\mathbf{x}}_{\epsilon}, \epsilon) = \hat{\mathbf{x}}_\epsilon\), which is exact by the boundary condition. Once the model is right at level \(t_n\), the loss makes it right at \(t_{n+1}\), and so on up to \(T\). The correct answer is passed up the ladder of noise levels, one rung per teacher step. This is bootstrapping, the same idea as temporal-difference learning in RL, where the target is the model’s own (slowly updated) prediction at the next state (session 65).
The same argument shows the limit: CD learns the end point of the teacher’s discretized ODE, not of the true ODE. If the model is perfect, a 1-step consistency sample is exactly as good as running the teacher solver for all \(N - 1\) steps. The paper bounds the error by the solver’s local error. [Certain] So \(N\) and the teacher solver set a floor on quality, just as in Figure 2 the 64-step teacher set the floor.
- input: teacher denoiser \(D\); ODE solver \(\Phi\) (Euler or Heun); levels \(t_1 = \epsilon \lt \dots \lt t_N = T\); EMA rate \(\mu\)
- \(\theta^- \leftarrow \theta\)
- repeat:
- \(\mathbf{x}_0 \sim\) data; \(n \sim \text{Uniform}\{1, \dots, N - 1\}\); \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\); \(\mathbf{x} \leftarrow \mathbf{x}_0 + t_{n+1} \mathbf{z}\)
- \(\hat{\mathbf{x}} \leftarrow\) one solver step from \((\mathbf{x}, t_{n+1})\) to \(t_n\) with the teacher // 1 or 2 teacher calls, no gradients
- \(\mathcal{L} \leftarrow \lambda(t_n)\, d\big(f_\theta(\mathbf{x}, t_{n+1}),\ f_{\theta^-}(\hat{\mathbf{x}}, t_n)\big)\) // (230.8), \(f\) as in (230.7)
- \(\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}\)
- \(\theta^- \leftarrow \text{stopgrad}(\mu\, \theta^- + (1 - \mu)\, \theta)\) // slow target
Figure 4 runs Algorithm 2 for real, in your browser, on the 1-D toy. The teacher is the exact denoiser of the mixture with one Euler or Heun step. The model \(F_\theta\) is tiny: at each level \(t_2, \dots, t_N\) it is a table of 16 numbers (at 16 quantiles of the noisy data, straight lines between them), plugged into (230.7). All tables start at zero, so at first \(f_\theta \approx c_{\text{skip}}\, x\), which is almost 0 at high noise. Training is plain SGD on one sample per step, with weight \(\lambda = 1/c_{\text{out}}(t_{n+1})^2\) and learning rate \(\eta = 0.2\), and a target network updated by EMA. Watch two things: the boxed formulas with the live numbers of the current sample, and the per-level error bars, which fall from left to right as the correct answer is passed up the ladder.
- \(x_0 \sim\) data; \(n \sim \text{Uniform}\{1, \dots, N-1\}\); \(z \sim \mathcal{N}(0, 1)\)
- \(x \leftarrow x_0 + t_{n+1}\, z\) // noisy point
- \(\hat{x} \leftarrow x + (t_n - t_{n+1})\, \Phi(x, t_{n+1})\) // one teacher step, no gradient
- \(a \leftarrow f_\theta(x, t_{n+1}) = c_{\text{skip}} x + c_{\text{out}} F_\theta(x, t_{n+1})\) // online
- \(b \leftarrow f_{\theta^-}(\hat{x}, t_n)\) // target network; \(b = \hat{x}\) when \(n = 1\)
- \(\mathcal{L} \leftarrow \lambda\, (a - b)^2\), \(\lambda = 1/c_{\text{out}}(t_{n+1})^2\) // (230.8)
- \(\theta \leftarrow \theta - \eta\, \nabla_\theta \mathcal{L}\) // only 2 table entries move
- \(\theta^- \leftarrow \mu\, \theta^- + (1 - \mu)\, \theta\) // slow target
How to read Figure 4
Press Step a few times at slow speed: each press runs one line of the box, and the right panel shows that line’s formula with the numbers of this sample. Then raise the speed: Play now runs many SGD steps per frame. With the defaults (\(N = 10\), Heun, \(\mu = 0.9\)) the one-step error is still 1.37 after 1,000 steps (it starts at 1.44), because the top levels have not learned anything yet: their targets are still wrong. Meanwhile the error bars of the low levels fall first, then the next ones, from left to right. At 4,000 steps \(W_1\) is 0.14, and later it settles near 0.17 to 0.18, close to the dashed teacher-chain value 0.167: CD cannot beat the solver it distills. Try \(N = 18\) with the Euler teacher: more rungs, so the information needs longer to climb (\(W_1\) is still 0.77 at 8,000 steps), but the floor is lower (0.155). Try \(\mu = 0.99\): the slower target delays learning (1.10 instead of 0.80 at 2,000 steps).
Consistency training (CT): no teacher at all
The teacher’s only job in (230.8) is to give a point \(\hat{\mathbf{x}}_{t_n}\) on the same trajectory as \(\mathbf{x}_{t_{n+1}}\). Consistency training replaces it with a cheap estimate: use the same training image and the same noise \(\mathbf{z}\) at both levels:
This is justified because the score has an unbiased estimate from a single training pair, \(\nabla \log p_t(\mathbf{x}_t) = -\E[(\mathbf{x}_t - \mathbf{x}_0)/t^2 \mid \mathbf{x}_t]\); as \(N\) grows the CT loss matches the CD loss with an Euler teacher. [Certain] CT turns consistency models into a stand-alone family of generative models, not just a distillation trick. In the first paper CT was clearly worse than CD: one-step FID 8.70 vs 3.55 on CIFAR-10, and 13.0 vs 6.20 on ImageNet 64 × 64. [Certain]
Later work closed the gap. Improved consistency training (iCT; Song and Dhariwal, 2023) removed the EMA from the teacher network (they found it was a flaw in the theory), replaced LPIPS with a Pseudo-Huber loss, used a lognormal noise schedule, and doubled \(N\) at set intervals during training; it reports one-step FID 2.51 on CIFAR-10 and 3.25 on ImageNet 64 × 64, and 2.24 and 2.77 with two steps. [Certain] sCM (Lu and Song, 2024) makes training work in continuous time (no discrete \(N\)), scales to 1.5B parameters, and reports two-step FID 2.06 on CIFAR-10, 1.48 on ImageNet 64 × 64 and 1.88 on ImageNet 512 × 512, within 10% of the best diffusion models. [Certain]
Multistep consistency sampling
One step gives one sample. If you have budget for more, re-noise and denoise again:
The extra noise is fresh, so this is not an ODE solver; it is “jump to the end, step back up to a middle noise level, jump to the end again”. The second jump starts from a lower noise level, where the model is more accurate, and it can fix details of the first sample. The paper chooses the \(\tau_k\) greedily with a search on FID. [Certain] On CIFAR-10, CD improves from FID 3.55 (1 step) to 2.93 (2 steps); on ImageNet 64 × 64 from 6.20 to 4.70. Progressive distillation on the same tables: 8.34 and 5.58 on CIFAR-10, 15.39 and 8.95 on ImageNet 64 × 64. [Certain]
Figure 5 animates (230.10) on the 1-D toy. The model is a fully trained CD model (the fixed point of (230.8), the same as in Figure 6). 400 starting noises (the 400 quantiles of \(\mathcal{N}(0, T^2)\)) jump to the clean side in one call; then all of them are re-noised with fresh noise and jump again. The re-noise levels are \(\tau = 2, 0.6, 0.15\), each rounded to the nearest level of the model’s grid. The fresh noise is drawn with a fixed seed and is stratified (a shuffled set of normal quantiles), so that 400 points give a stable error.
- \(\mathbf{x}_T \sim \mathcal{N}(0, T^2)\) // 400 starting noises
- \(\mathbf{x} \leftarrow f_\theta(\mathbf{x}_T, T)\) // one jump: this is 1-step sampling
- for \(k = 1, \dots, K - 1\):
- \(\mathbf{z}_k \sim \mathcal{N}(0, 1)\) // fresh noise
- \(\hat{\mathbf{x}} \leftarrow \mathbf{x} + \sqrt{\tau_k^2 - \epsilon^2}\, \mathbf{z}_k\) // step back up to \(\tau_k\)
- \(\mathbf{x} \leftarrow f_\theta(\hat{\mathbf{x}}, \tau_k)\) // jump again, from lower noise
- return \(\mathbf{x}\) // \(K\) network calls in total
How to read Figure 5
Press Play. All samples start at the right (\(\sigma = 80\)) and jump to the left in one call. With \(N = 10\) and the Heun teacher, this one-step sample has \(W_1 = 0.167\): the modes are there but a bit too wide. Then the samples spread out again to \(\tau_1 = 1.50\) (the grid level nearest to 2) and jump back: \(W_1 = 0.128\). Two more rounds, at \(\tau = 0.47\) and \(0.117\), give 0.084 and 0.073. Each extra call costs one more network evaluation and fixes part of the error that the first long jump left. The tracked sample shows the numbers of (230.10) at each line. With \(N = 18\) and Heun, one step is already good (0.040). The second step is worse (0.065): the jump back from \(\tau = 1.92\) brings back some error. Only 3 or 4 steps go below one step (0.036 and 0.032). Extra steps help most when one step is weak, and \(\tau\) must be chosen with care (the paper searches for it).
Figure 6 runs consistency distillation on the 1-D toy. The model at each level is again a lookup table (96 points), trained level by level so that (230.8) is exactly zero at its points, which is the fixed point that CD training moves toward.
How to read Figure 6
At the default (Heun, \(N = 18\), the paper’s CIFAR-10 setting), one step gives \(W_1 = 0.0402\), about 35 times better than one DDIM step (1.41). Switch to 2-step sampling: with \(\tau \approx 1.09\) the error drops to 0.0237. Switch the teacher to Euler: one step gives 0.153, close to what a 17-step DDIM sampler gives, as the theory says. Now slide \(N\): the error falls about as \(1/N\) with Euler and \(1/N^2\) with Heun (Euler 0.071 and Heun 0.011 at \(N = 40\)). But at \(N = 3\) Heun is far worse (5.99 vs 0.78): its second evaluation lands at a point where the slope says little about a step that long. Two-step sampling helps most when one step is weak; try \(\tau\) near 1 to 1.5.
From toy models to products
Latent Consistency Models (LCM) and LCM-LoRA
LCM (Luo et al., 2023) applies consistency distillation to Stable Diffusion in its latent space (session 228). Three changes make it practical:
- Guidance inside the ODE. The teacher’s direction uses the CFG-combined noise prediction, so the student learns the guided ODE and needs no second call at sampling time:
- Read the symbols
- \(\mathbf{c}, \varnothing\)
- the text condition and the empty prompt
- \(\omega\)
- the guidance scale; during training it is drawn from \([2, 14]\) and given to the student as an extra input (with a zero-initialized embedding), so it can be chosen at sampling time; the tables use \(\omega = 8\) [Certain]
- Skipping steps. Instead of enforcing consistency between adjacent levels of the 1000-step schedule, LCM enforces it between \(t_{n+k}\) and \(t_n\) with \(k = 20\), using a DDIM teacher step over that gap. This makes the bootstrapping chain 50 links long instead of 1000, which speeds up training. [Certain]
- Cheap training. The authors report 32 A100 GPU hours for a 768 × 768, 2-to-4-step model. [Certain]
LCM-LoRA (Luo et al., 2023) trains the consistency distillation as a LoRA adapter (session 57) for SD-v1.5, SSD-1B and SDXL. The authors report that the same adapter can be plugged into other fine-tuned versions of the base model, without more training, and call it a “universal acceleration module”. [Certain] This matters for products: one small file turns a whole ecosystem of style fine-tunes into 4-step models.
Adversarial and distribution-matching distillation
Trajectory methods (progressive distillation, CD, LCM) copy the teacher’s ODE map. At 1 step this map is very hard to fit, and the student’s errors show up as blur. A second family drops the trajectory and asks only that the student’s samples look like real data or like the teacher’s distribution:
- Adversarial Diffusion Distillation (ADD; Sauer et al., 2023) trains the student with a GAN loss plus a score-distillation loss from the teacher:
The discriminator is a frozen DINOv2 ViT-S feature network with small trainable heads; the student uses 4 noise levels and can sample in 1 to 4 steps. ADD-XL (3.1B parameters, distilled from SDXL) beats LCM-XL at 4 steps with a single step in their human study, and beats SDXL at 50 steps with 4 steps in most comparisons. [Certain] The released SDXL-Turbo is an ADD-distilled SDXL that works at 512 × 512 with guidance_scale=0.0. [Certain]
- Distribution Matching Distillation (DMD; Yin et al., 2023) makes a one-step generator \(G_\theta\) whose output distribution has low KL divergence to the teacher’s. The gradient needs only two score functions: the real one (the frozen teacher) and a “fake” one, a second diffusion model trained online on the generator’s outputs:
The difference of scores pushes each generated sample away from where the generator is too dense and toward where the real data is denser. DMD adds a regression loss on a set of teacher (noise, image) pairs to keep the large-scale structure. It reports FID 2.62 on ImageNet 64 × 64 and 11.49 on zero-shot COCO-30k, and 20 images per second with FP16 on modern hardware. [Certain]
- Later variants. SDXL-Lightning (Lin et al., 2024) combines progressive and adversarial distillation for 1024 px SDXL and releases LoRA and full weights. [Certain] LADD (Sauer et al., 2024) moves the discriminator into latent space, using features of a pretrained latent diffusion model instead of DINOv2, and makes SD3-Turbo (from the 8B SD3) with 4 unguided steps. [Certain] FLUX.1 schnell, a 12B rectified-flow transformer under Apache-2.0, is “trained using latent adversarial diffusion distillation” and makes images in 1 to 4 steps; its example code uses 4 steps and
guidance_scale=0.0. [Certain]
Straighter paths
The third route makes the path itself easier. If trajectories were straight lines, one Euler step would be exact. Rectified flow (session 229) “reflows” a trained model on its own (noise, image) pairs to straighten paths, then distills. InstaFlow (Liu et al., 2023) used this on Stable Diffusion: one step, FID 23.3 on MS COCO 2017-5k against 37.2 for progressive distillation, about 0.09 s per image, for about 199 A100 GPU-days of training. [Certain]
Quality vs steps vs latency
Every few-step method trades something. The LCM paper gives a clean comparison on Stable Diffusion with the same guidance scale (\(\omega = 8\)): FID (image quality and variety against real images, lower is better) and CLIP score (how well the image matches the prompt, higher is better) at 1, 2, 4 and 8 steps, for two training-free solvers, a guided progressive distillation baseline and LCM. Figure 8 plots their Tables 1 and 2; the evaluation uses 30K images from 10K prompts of LAION-Aesthetics. [Certain]
How to read Figure 8
At 1 and 2 steps, the solvers are useless (FID 183 and 81 for DDIM at 512 px) and LCM is far ahead (35.4 and 13.3). At 4 steps LCM is still about 2 times better than DDIM (11.10 vs 22.38). At 8 steps the solvers almost catch up, and in CLIP score they pass LCM (29.29 for DDIM vs 28.84). At 768 px, LCM’s FID gets slightly worse from 4 to 8 steps (13.53 to 14.97). Distilled models are built for a few steps; more steps do not always help them.
How to choose
- Quality per step is not quality. Distilled models give much better images at 1 to 4 steps, but the teacher at 25 to 50 steps is usually still the best, especially in fine detail, text rendering and variety. [Likely] Keep the teacher for “high quality” mode and use the student for previews and real-time modes.
- Diversity. Trajectory methods keep the teacher’s map from noise to image, so different seeds give different images. Adversarial and mode-seeking objectives can reduce variety (DMD explicitly adds a regression loss to keep the structure of the teacher’s outputs). Measure diversity, not just FID or a preference score. [Likely]
- Guidance is frozen in. LCM learns a range of \(\omega\); SDXL-Turbo and FLUX.1 schnell are used with guidance off. Negative prompts and strong guidance edits, which need the unconditional branch, may not work the same way. [Likely]
- Steps you can still choose. Consistency models and LCMs let you pick 1, 2, 4 or 8 steps at sampling time. Progressive distillation gives a fixed grid per student (but you can keep the students from all rounds).
- Latency is not NFE. With 1 to 4 calls, the VAE decoder, text encoder, memory transfers and Python overhead matter (Figure 1). Profile the full pipeline; CUDA graphs and compilation help here (session 127).
- Video multiplies everything. One call of a video model processes all frames of a clip, so a 50-step video sample is very expensive, and few-step distillation is even more valuable there (session 231). Temporal consistency must survive the distillation: a student that is sharp per frame but flickers is a failure. [Likely]
Few-step distillation in PyTorch
A complete, deterministic example on the 1-D mixture. The teacher is the exact denoiser, so all remaining error comes from the student network and the method. Part 1 trains a consistency model by consistency distillation (Heun teacher, \(N = 18\), squared L2 distance, EMA target) and samples it with 1 and 2 steps. Part 2 runs four rounds of progressive distillation, 16 → 8 → 4 → 2 → 1 steps, with a small MLP student. It runs in about 15 seconds on a laptop CPU.
import math, torch, torch.nn as nn torch.manual_seed(0) # toy data: a 1-D mixture of three Gaussians (same as the Diffusion 2 session) W = torch.tensor([0.35, 0.45, 0.20]) MU = torch.tensor([-2.0, 0.5, 2.5]) SD = torch.tensor([0.3, 0.4, 0.2]) SMIN, SMAX, RHO = 0.002, 80.0, 7.0 SIG_D = 0.5 # sigma_data in c_skip, c_out (the CM default) def sample_data(n): c = torch.multinomial(W, n, replacement=True) return MU[c] + SD[c] * torch.randn(n) def D_teacher(x, s): """The exact denoiser E[x0 | x_sigma] of the mixture: a perfect teacher.""" v = SD**2 + s[:, None]**2 logw = torch.log(W) - 0.5 * torch.log(v) - 0.5 * (x[:, None] - MU)**2 / v r = torch.softmax(logw, dim=1) return (r * (MU + SD**2 / v * (x[:, None] - MU))).sum(1) def karras(N): # sigma grid, ascending: t_1 = eps ... t_N = T i = torch.arange(N, dtype=torch.float64) return ((SMIN**(1/RHO) + i/(N-1) * (SMAX**(1/RHO) - SMIN**(1/RHO)))**RHO).float() def heun(x, a, b): # one Heun step of the PF-ODE, sigma a -> b d = (x - D_teacher(x, a)) / a xe = x + (b - a) * d d2 = (xe - D_teacher(xe, b)) / b return x + (b - a) * (d + d2) / 2 class F(nn.Module): # tiny network F_theta(x, sigma) def __init__(s, h=128): super().__init__() s.net = nn.Sequential(nn.Linear(2, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 1)) def forward(s, x, sig): c_in = 1 / torch.sqrt(sig**2 + SIG_D**2) return s.net(torch.stack([c_in * x, torch.log(sig) / 4], 1)).squeeze(1) def f_cm(model, x, sig): # formula (230.6): boundary f(x, eps) = x built in c_skip = SIG_D**2 / ((sig - SMIN)**2 + SIG_D**2) c_out = SIG_D * (sig - SMIN) / torch.sqrt(SIG_D**2 + sig**2) return c_skip * x + c_out * model(x, sig) # ---- consistency distillation, formula (230.7) ---- N, MU_EMA, STEPS, B = 18, 0.95, 4000, 512 t = karras(N) model = F(); target = F(); target.load_state_dict(model.state_dict()) for p in target.parameters(): p.requires_grad_(False) opt = torch.optim.Adam(model.parameters(), lr=1e-3) for it in range(STEPS): x0 = sample_data(B) n = torch.randint(0, N - 1, (B,)) # pair (t_n, t_{n+1}) a, b = t[n + 1], t[n] xa = x0 + a * torch.randn(B) # a point on the noisy marginal at t_{n+1} with torch.no_grad(): xb = heun(xa, a, b) # one teacher ODE step down to t_n tgt = f_cm(target, xb, b) # f_{theta^-}(x_hat_{t_n}, t_n) loss = ((f_cm(model, xa, a) - tgt)**2).mean() # squared L2 distance, lambda = 1 opt.zero_grad(); loss.backward(); opt.step() with torch.no_grad(): # EMA target network for pt, pm in zip(target.parameters(), model.parameters()): pt.mul_(MU_EMA).add_(pm, alpha=1 - MU_EMA) # ---- sampling and evaluation (1-D Wasserstein-1 against fresh data) ---- def w1(x, ref): return (x.sort().values - ref.sort().values).abs().mean().item() g = torch.Generator().manual_seed(1) M = 20000 ref = sample_data(M) zT = SMAX * torch.randn(M, generator=g) with torch.no_grad(): one = f_cm(model, zT, torch.full((M,), SMAX)) # 1 NFE tau = 1.0 xt = one + math.sqrt(tau**2 - SMIN**2) * torch.randn(M, generator=g) two = f_cm(model, xt, torch.full((M,), tau)) # 2 NFE, formula (230.9) x = zT.clone() # DDIM with 2 NFE, same noise for a_, b_ in [(SMAX, 1.0), (1.0, 0.0)]: d = D_teacher(x, torch.full((M,), a_)) x = d + (b_ / a_) * (x - d) print(f"CD 1 step W1 = {w1(one, ref):.3f}") print(f"CD 2 steps W1 = {w1(two, ref):.3f}") print(f"DDIM 2 steps (perfect denoiser) W1 = {w1(x, ref):.3f}") print(f"data vs data floor W1 = {w1(sample_data(M), ref):.3f}") # ---- progressive distillation, formulas (230.3) and (230.4): 16 -> 8 -> 4 -> 2 -> 1 steps ---- def karras_desc(N): # sampling grid, descending, with sigma_N = 0 return torch.cat([karras(N).flip(0), torch.zeros(1)]) def ddim(x, a, b, denoise): # formula (230.2) in sigma form d = denoise(x, a) return d + (b / a) * (x - d) class Student(nn.Module): # D_eta(x; sigma) = c_skip x + c_out F(x, sigma) def __init__(s): super().__init__(); s.F = F() def forward(s, x, sig): c_skip = SIG_D**2 / (sig**2 + SIG_D**2) c_out = sig * SIG_D / torch.sqrt(sig**2 + SIG_D**2) return c_skip * x + c_out * s.F(x, sig) def sample(denoise, grid, z): x = z.clone() for i in range(len(grid) - 1): x = ddim(x, torch.full_like(x, grid[i]), torch.full_like(x, grid[i + 1]), denoise) return x teacher, grid = D_teacher, karras_desc(16) # round 0: the exact denoiser with 16 DDIM steps with torch.no_grad(): print(f"teacher 16-step W1 = {w1(sample(teacher, grid, zT), ref):.3f}") student = Student() # round 1 starts from scratch; later rounds start from the last student for rnd in range(4): new_grid = grid[::2] # every other sigma: half the steps opt = torch.optim.Adam(student.parameters(), lr=1e-3) for it in range(1500): j = torch.randint(0, len(new_grid) - 1, (B,)) a, b = new_grid[j], new_grid[j + 1] mid = grid[2 * j + 1] # the teacher's in-between sigma xa = sample_data(B) + a * torch.randn(B) with torch.no_grad(): # two teacher DDIM steps a -> mid -> b xb = ddim(ddim(xa, a, mid, teacher), mid, b, teacher) ok = b > 0 x_tilde = torch.where(ok, (a * xb - b * xa) / (a - b), xb) # (230.3) w = torch.clamp(1 / a**2, min=1.0) # truncated SNR weight, max(SNR, 1) loss = (w * (student(xa, a) - x_tilde)**2).mean() # (230.4) opt.zero_grad(); loss.backward(); opt.step() frozen = Student(); frozen.load_state_dict(student.state_dict()) for p in frozen.parameters(): p.requires_grad_(False) teacher, grid = frozen, new_grid # the student becomes the next teacher with torch.no_grad(): print(f"student {len(grid)-1:2d}-step W1 = {w1(sample(teacher, grid, zT), ref):.3f}" f" DDIM {len(grid)-1:2d}-step, perfect denoiser: {w1(sample(D_teacher, grid, zT), ref):.3f}")
Output of this script (PyTorch 2, CPU):
CD 1 step W1 = 0.065 CD 2 steps W1 = 0.040 DDIM 2 steps (perfect denoiser) W1 = 0.702 data vs data floor W1 = 0.011 teacher 16-step W1 = 0.169 student 8-step W1 = 0.170 DDIM 8-step, perfect denoiser: 0.344 student 4-step W1 = 0.170 DDIM 4-step, perfect denoiser: 0.609 student 2-step W1 = 0.167 DDIM 2-step, perfect denoiser: 0.684 student 1-step W1 = 0.164 DDIM 1-step, perfect denoiser: 1.403
Read it the same way as the figures. The consistency model gives \(W_1 = 0.065\) in one call and 0.040 in two, while two DDIM steps with the perfect denoiser give 0.702. The distilled students keep the 16-step teacher’s quality (0.169) all the way down to one step (0.164; the small changes are training noise), while plain DDIM on the same grids degrades to 1.403. The floor 0.011 is the error between two independent sets of 20,000 data samples, so differences smaller than that mean nothing.
Where this is used
| Setting | Method | What matters |
|---|---|---|
| Real-time text-to-image (typing-speed previews, live drawing-to-image) | ADD (SDXL-Turbo), LCM, LCM-LoRA: 1 to 4 steps | Total pipeline latency, including decoder (Figure 1) |
| Open-weight fast image models | FLUX.1 schnell (LADD, 1 to 4 steps, Apache-2.0); SD3-Turbo (LADD, 4 steps); SDXL-Lightning (LoRA or full weights) | Guidance off; quality close to the teacher at 4 steps |
| Speeding up a community of fine-tunes | LCM-LoRA as a plug-in adapter | One adapter, many styles; check quality per style |
| Image editing, inpainting, colorization, super-resolution | Consistency models support these zero-shot, without task-specific training [Certain]; the known pixels are put back between steps [Likely] | Few steps make interactive editing possible |
| Video generation | Consistency and adversarial distillation of video diffusion models into few-step samplers [Likely] | Per-call cost is huge; must keep frames consistent over time |
| Synthetic data for video tasks (action recognition, retrieval, key-moment detection) | Few-step generators make large synthetic sets affordable [Speculation] | Diversity loss of distilled models can bias the data |
| Robot policies and VLA models that sample actions with diffusion | Consistency-style or distilled policies for fast control loops [Likely] | Control rate (for example 10 to 50 Hz) sets the step budget |
| Generated images inside a VLM or an any-to-any model | A few-step decoder after the language model (session 232) | Image latency adds to the text answer latency |
Common mistakes
- Expecting a solver to fix 1 to 4 steps. With a pretrained, undistilled model, 1 DDIM step returns the posterior mean: a blur, whatever the solver. Below about 8 steps you need a distilled or consistency model.
- Using a distilled model with the teacher’s sampler settings. LCM needs its own scheduler; Turbo and schnell models expect guidance off. Running CFG on top of a guidance-distilled model applies guidance twice and over-saturates images. [Likely]
- More steps on a few-step model. Distilled models are tuned for their step counts; going from 4 to 8 often brings little, and sometimes hurts (Figure 8, 768 px).
- \(\varepsilon\)-prediction for a 1-step student. At the highest noise level the clean guess from \(\hat\varepsilon\) divides by a near-zero \(\alpha_t\). Use \(\mathbf{x}\)-, v- or EDM-style skip parameterizations (230.7).
- Forgetting the boundary condition. Without \(c_{\text{skip}}(\epsilon) = 1\), \(c_{\text{out}}(\epsilon) = 0\), the model can satisfy self-consistency with a constant output (all points map to the same image). The boundary condition is what ties the chain to real data.
- Backpropagating through the target. The target \(f_{\theta^-}(\hat{\mathbf{x}}_{t_n}, t_n)\) in (230.8) must be computed without gradients; otherwise the model can lower the loss by moving the target and training becomes unstable.
- Judging only by FID or a preference score. Check diversity across seeds, prompt following, text rendering and, for video, temporal flicker. One-step models can look great on average and fail on the long tail.
- Measuring only the denoiser loop. At 1 to 4 steps the decoder and text encoder can be half the time; report end-to-end latency.
Interview questions
Why is diffusion sampling slow, and what are the main families of fixes?basic
Why does one DDIM step from pure noise give a blurry image, even with a perfect network?core
Explain progressive distillation. What is the student’s target?core
Why did progressive distillation need new parameterizations such as v-prediction?advanced
What is a consistency model? Define self-consistency and the boundary condition.basic
Consistency distillation vs consistency training: what is the difference, and what does each cost?core
In consistency distillation, why use a target network \(\theta^-\) with stop-gradient? What stops the model from outputting a constant?advanced
Why can 2-step consistency sampling beat 1 step, if the model already predicts the end point?core
How does LCM deal with classifier-free guidance, and what is LCM-LoRA?core
Compare trajectory distillation (CD, LCM) with adversarial or distribution-matching distillation (ADD, DMD).advanced
Your text-to-image service takes about 5 s per image (50 steps with CFG). Product wants under 300 ms. Walk through your plan.core
What changes when you distill a video diffusion model instead of an image model?advanced
Exercises
- A progressive distillation run starts from a 1024-step teacher. How many rounds to reach 4 steps? To reach 1 step?
- Two teacher DDIM steps take \(\mathbf{x}_a = 0.8\) at \(a = 1\) to \(\mathbf{x}_b'' = 0.3\) at \(b = 0.5\). Compute the distillation target \(\tilde{\mathbf{x}}\) and check it.
- Compute \(c_{\text{skip}}\) and \(c_{\text{out}}\) of (230.7) with \(\sigma_{\text{data}} = 0.5\) and \(\epsilon = 0.002\) at \(t = \epsilon\) and \(t = 80\). What do they tell you?
- A model runs 30 steps with CFG, 40 ms per call and 80 ms fixed time. What is the latency? What is it for a 4-step guidance-distilled student, and what is the speed-up?
- In Figure 6, Heun at \(N = 3\) is much worse than Euler at \(N = 3\), but much better at \(N = 18\). Explain both.
- Show that in 1-D, \(f(x, \sigma) = Q_0(F_\sigma(x))\) satisfies self-consistency and the boundary condition.
Answerscheck
Key takeaways
- Latency is \(T_{\text{fixed}} + N c\, t_{\text{net}}\) (230.1). Few-step methods cut \(N\) and remove the CFG factor \(c\); then fixed costs dominate.
- One DDIM step (230.2) returns the posterior mean, a blur, even with a perfect network. Few-step generation needs a network trained for big steps.
- Progressive distillation (230.3, 230.4) halves the steps per round by matching two teacher steps with one; quality stays near the teacher if the student has the capacity (Figure 2). Use x-, v- or skip parameterizations, not \(\varepsilon\).
- Consistency models learn \(f(\mathbf{x}_t, t)\) = the end of the ODE path (230.5), with the boundary condition built in (230.7); CD bootstraps with one teacher step and a stop-gradient target (230.8), CT uses shared noise instead of a teacher (230.9). Multistep sampling (230.10) trades calls for quality.
- Products: LCM and LCM-LoRA (guidance in the ODE, 230.11), ADD and LADD (adversarial, 230.12), DMD (score difference, 230.13), and straight rectified-flow paths. FLUX.1 schnell and SDXL-Turbo run in 1 to 4 steps with guidance off.
- Judge few-step models on quality, diversity, prompt following and end-to-end latency, and for video on temporal coherence.
Key formulas
References for this session
- Salimans & Ho (2022). Progressive Distillation for Fast Sampling of Diffusion Models
- Song et al. (2023). Consistency Models
- Song et al. (2020). Denoising Diffusion Implicit Models
- Ho & Salimans (2022). Classifier-Free Diffusion Guidance
- Liu et al. (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- Karras, Aittala, Aila & Laine (2022). Elucidating the Design Space of Diffusion-Based Generative Models
- Song & Dhariwal (2023). Improved Techniques for Training Consistency Models
- Lu & Song (2024). Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models
- Luo, Tan, Huang, Li & Zhao (2023). Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
- Luo, Tan, Patil, Gu, von Platen, Passos, Huang, Li & Zhao (2023). LCM-LoRA: A Universal Stable-Diffusion Acceleration Module
- Meng, Rombach, Gao, Kingma, Ermon, Ho & Salimans (2023). On Distillation of Guided Diffusion Models
- Sauer, Lorenz, Blattmann & Rombach (2023). Adversarial Diffusion Distillation
- Sauer, Boesel, Dockhorn, Blattmann, Esser & Rombach (2024). Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation
- Yin, Gharbi, Zhang, Shechtman, Durand, Freeman & Park (2024). One-step Diffusion with Distribution Matching Distillation
- Lin, Wang & Yang (2024). SDXL-Lightning: Progressive Adversarial Diffusion Distillation
- Liu, Zhang, Ma, Peng & Liu (2023). InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation
- Black Forest Labs (2024). FLUX.1 [schnell] model card