LLM & VLM courseBlog
227Track 9: Generative models · session 6 of 12

Guidance and conditioning: classifier guidance, classifier-free guidance, ControlNet

A diffusion model that only learned p(x | prompt) follows the prompt weakly. This session shows how the prompt enters the network (cross-attention), how classifier guidance and classifier-free guidance push samples toward the prompt, why a large guidance scale gives sharp but less varied images, and how ControlNet adds spatial control (edges, pose, depth) to a frozen model with zero-initialized convolutions.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Diffusion 1 and 2 (sessions 225 and 226: noise prediction, the score, DDIM), attention (Q, K, V), Bayes' rule; basic PyTorch

What you will learn

  • How text conditioning enters a diffusion U-Net through cross-attention: queries from image tokens, keys and values from text tokens, with the real Stable Diffusion 1.5 shapes
  • Bayes' rule for scores, and how classifier guidance adds the gradient of a noisy classifier to the noise prediction
  • Classifier-free guidance: one network trained with label dropout, two forward passes per step, and the formula eps_u + w (eps_c - eps_u)
  • The two conventions for the guidance scale (Ho and Salimans w versus the Stable Diffusion scale) and how to convert between them
  • Why high guidance gives sharp but less varied images, shown exactly on a 1-D toy and with the published FID, IS, precision and recall numbers
  • What goes wrong at very high guidance (saturation, error amplification) and the common fixes: guidance rescale, thresholding, limited intervals
  • ControlNet: a locked model, a trainable copy, and zero convolutions that make training start exactly at the pretrained model

Why this matters

Type “a red car on a beach” into Stable Diffusion, and the model returns a red car on a beach. Two separate mechanisms make that happen. Conditioning lets the network read the prompt. Guidance makes the sampler obey it strongly. Without guidance, the same model gives blurry, off-topic images much of the time.

In session 225 and session 226 we learned to sample from \(p(\mathbf{x})\): any image that looks like training data. Real products need \(p(\mathbf{x} \mid \mathbf{y})\), where \(\mathbf{y}\) is a class label, a text prompt, an edge map, a pose skeleton or a depth map. This session answers three questions:

  1. How does the condition enter the network? For text: cross-attention from image tokens to text tokens.
  2. How do we make samples follow the condition strongly? Classifier guidance (Dhariwal and Nichol, 2021) and classifier-free guidance (CFG) (Ho and Salimans, 2022). CFG is in almost every text-to-image and text-to-video model. The “guidance scale” slider in every image app is the CFG scale.
  3. How do we add a spatial condition to a big pretrained model without breaking it? ControlNet (Zhang, Rao and Agrawala, 2023): a trainable copy of the encoder, connected through convolutions initialized to zero.

The trade-off is the heart of the topic. Guidance moves probability mass toward samples that are “very clearly” the condition. That raises sample quality and prompt alignment, but it lowers diversity. In the original classifier-guidance paper, a scale of 10 on a conditional ImageNet model raised precision from 0.69 to 0.88 and lowered recall from 0.63 to 0.32. [Certain] You will see exactly why, on a toy problem you can solve by hand.

Conditioning: how the prompt gets into the network

A conditional diffusion model is a noise predictor with one extra input: \(\epsilon_\theta(\mathbf{x}_t, t, \mathbf{y})\). It is trained with the same loss as before, only now each training image comes with its label or caption \(\mathbf{y}\). The question is how \(\mathbf{y}\) is fed in. There are three common ways:

  • Add it to the time embedding. A class label becomes a learned vector, added to the vector that encodes \(t\). Every residual block already receives the time embedding, so every block now also “knows” the class. This is how class-conditional ImageNet models work. [Certain]
  • Concatenate it to the input. For spatial conditions with the same size as the image (a low-resolution image for super-resolution, a mask for inpainting), stack them as extra input channels. The Stable Diffusion inpainting model has 5 extra input channels: 4 for the encoded masked image and 1 for the mask. [Certain]
  • Cross-attention. For text, which is a sequence of variable meaning, the image features attend to the text tokens. This is the main mechanism in Stable Diffusion 1.x, 2.x and SDXL. [Certain]

Cross-attention

Recall attention from session 20: queries ask, keys answer “how relevant am I”, values carry the content. In self-attention, Q, K and V all come from the same sequence. In cross-attention, the queries come from one sequence and the keys and values from another. In the diffusion U-Net, the queries come from the image features, and the keys and values come from the text encoder output:

\[ \mathbf{Q} = \mathbf{H} W_Q, \quad \mathbf{K} = \tau(\mathbf{y}) W_K, \quad \mathbf{V} = \tau(\mathbf{y}) W_V, \quad \operatorname{CrossAttn}(\mathbf{H}, \mathbf{y}) = \softmax\!\left( \frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d}} \right) \mathbf{V} \tag{227.1} \]
Read the symbols
\(\mathbf{H}\)
the image feature map flattened into tokens: \(h \cdot w\) rows (one per spatial position), \(C\) columns (channels)
\(\tau(\mathbf{y})\)
the text encoder output for prompt \(\mathbf{y}\): one vector per text token
\(W_Q, W_K, W_V\)
learned projection matrices; \(W_K, W_V\) map the text width to the image width
\(d\)
the dimension of one attention head
result
for every image position, a weighted mix of text-token values, added back to \(\mathbf{H}\) through a residual connection

So each spatial position of the noisy image asks “which words are about me?” and pulls in their meaning. The position where the car should go attends to “red” and “car”; the bottom of the image attends to “beach”. This is also why attention maps from cross-attention layers can be used for editing (change one word, keep the layout).

noisy latent features 64 × 64 positions → 4,096 tokens 320 channels each (level 1) Q = HWQ4,096 × 320 prompt “a red car on a beach” text encoder τCLIP ViT-L/14, frozen 77 tokens × 768 K = τ(y) WK77 × 320 V = τ(y) WV77 × 320 softmax(QK⊤ / √d) Vper head: a 4,096 × 77 attention matrix + residual order inside each transformer block of the U-Net: self-attention → cross-attention → feed-forward
Figure 1. Cross-attention at the highest-resolution level of the Stable Diffusion 1.5 U-Net. Queries (blue) come from the 4,096 latent positions; keys (orange) and values (green) come from the 77 text tokens. The highlighted latent cell gets a mix of word meanings that is added back to its features.
Worked example: the shapes in Stable Diffusion 1.5

From the model’s unet/config.json: cross_attention_dim = 768, block_out_channels = [320, 640, 1280, 1280], latent sample_size = 64. [Certain] The text encoder is CLIP ViT-L/14 with a maximum of 77 tokens, so \(\tau(\mathbf{y})\) is \(77 \times 768\). [Certain]

At the first level the latent is \(64 \times 64 = 4{,}096\) positions with 320 channels, so \(\mathbf{Q}\) is \(4{,}096 \times 320\), and \(W_K, W_V\) are \(768 \times 320\). The config value attention_head_dim = 8 is used as the number of heads in this old config, so each head has \(320/8 = 40\) dimensions. [Likely] One head’s score matrix \(\mathbf{Q}\mathbf{K}^\top\) is \(4{,}096 \times 77 = 315{,}392\) numbers. The lower levels have \(32 \times 32 = 1{,}024\) and \(16 \times 16 = 256\) query tokens. Compare self-attention at level 1: \(4{,}096 \times 4{,}096\), which is 53 times bigger. Cross-attention is cheap because the text is short.

Larger models change only the numbers: SDXL has cross_attention_dim = 2048 because it concatenates two text encoders. [Certain] Diffusion transformers (next session, session 228) often replace cross-attention with joint attention over image and text tokens, or with adaptive LayerNorm for class labels and time.

Why conditioning alone is not enough

Train \(\epsilon_\theta(\mathbf{x}_t, t, \mathbf{y})\) perfectly and sample from it. You get exact samples of \(p(\mathbf{x} \mid \mathbf{y})\). That sounds like what we want, but in practice the samples look poor: many images match the prompt only loosely, and the overall quality is low. The original numbers show it. A class-conditional ImageNet 256 × 256 model without guidance had FID 10.94; the same model with classifier guidance (scale 1) had FID 4.59. [Certain] (FID, the Fréchet Inception Distance, compares statistics of generated and real images; lower is better.)

Two reasons. First, the model is not perfect, and the condition is a weak signal compared to the strong task of denoising, so the network partly ignores it. Second, even the true \(p(\mathbf{x} \mid \text{“dog”})\) contains many unclear, cluttered photos where the dog is small. People prefer the “typical, clear” dogs. In GANs this was solved with the truncation trick, and in language models with a temperature below 1 (session 79). Guidance is the diffusion version of low temperature: make the sample more clearly “of the class”.

The tool is Bayes’ rule. Take the log of \(p(\mathbf{x}_t \mid \mathbf{y}) = p(\mathbf{x}_t)\, p(\mathbf{y} \mid \mathbf{x}_t) / p(\mathbf{y})\) and the gradient with respect to \(\mathbf{x}_t\). The term \(p(\mathbf{y})\) does not depend on \(\mathbf{x}_t\), so it disappears:

\[ \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t \mid \mathbf{y}) = \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t) + \nabla_{\mathbf{x}_t} \log p(\mathbf{y} \mid \mathbf{x}_t) \tag{227.2} \]

The conditional score is the unconditional score plus the gradient of a classifier. The second term points toward images that the classifier finds more clearly of class \(\mathbf{y}\). If we multiply that term by a number larger than 1, we “over-condition”. Every method in this session is a variation of this idea.

Classifier guidance

From session 226: a noise predictor is a scaled score. For the variance-preserving process \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon\),

\[ \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t) \approx -\frac{\epsilon_\theta(\mathbf{x}_t, t)}{\sqrt{1 - \bar\alpha_t}} \tag{227.3} \]

Dhariwal and Nichol train a separate classifier \(p_\phi(\mathbf{y} \mid \mathbf{x}_t, t)\) on noisy images (the downsampling half of their U-Net, trained on the same noise distribution as the diffusion model). [Certain] Put (227.3) into (227.2), add a gradient scale \(s\), and turn the score back into a noise prediction:

\[ \hat{\epsilon}(\mathbf{x}_t) = \epsilon_\theta(\mathbf{x}_t, t) - s\,\sqrt{1 - \bar\alpha_t}\;\nabla_{\mathbf{x}_t} \log p_\phi(\mathbf{y} \mid \mathbf{x}_t, t) \tag{227.4} \]
Read the symbols
\(\epsilon_\theta\)
the diffusion model’s noise prediction (it may be unconditional or already class-conditional)
\(p_\phi(\mathbf{y} \mid \mathbf{x}_t, t)\)
a classifier that sees the noisy image and the time step
\(s\)
the gradient scale; \(s = 0\) is no guidance, \(s = 1\) is plain Bayes’ rule
\(\hat{\epsilon}\)
the guided noise prediction; DDIM or DDPM then runs as usual with \(\hat{\epsilon}\) in place of \(\epsilon_\theta\)

What distribution does this sample? Since \(s \nabla \log p(\mathbf{y} \mid \mathbf{x}) = \nabla \log p(\mathbf{y} \mid \mathbf{x})^s\), the guided score belongs to

\[ \tilde p_s(\mathbf{x}_t \mid \mathbf{y}) \;\propto\; p(\mathbf{x}_t)\; p_\phi(\mathbf{y} \mid \mathbf{x}_t)^{s} \tag{227.5} \]

Raising a probability to a power \(s \gt 1\) makes it sharper: regions where the classifier is 99% sure keep almost all their weight, regions where it is 60% sure lose most of it. The paper says it directly: a larger scale “focuses more on the modes of the classifier”. [Certain] At \(s = 1\) on an unconditional model, the classifier gave the right class about 50% probability on final samples, and the images did not look like the class; at \(s = 10\) the probabilities went to nearly 100%. [Certain]

ImageNet 256 × 256 (Dhariwal & Nichol, Table 4)FID ↓IS ↑Precision ↑Recall ↑
Unconditional, no guidance26.2139.700.610.63
Unconditional, scale 133.0332.920.560.65
Unconditional, scale 1012.0095.410.760.44
Conditional, no guidance10.94100.980.690.63
Conditional, scale 14.59186.700.820.52
Conditional, scale 109.11283.920.880.32

Read the table column by column. Precision (the share of generated images that fall inside the real-image distribution, a fidelity measure) goes up with the scale. Recall (the share of the real distribution that the generated images cover, a diversity measure) goes down. IS (Inception Score, high when each image is confidently one class and the classes are varied) rises a lot. FID, which punishes both bad images and missing diversity, has its best value at a moderate scale. [Certain]

The problems with classifier guidance

  • You need a second model, trained on noisy images at every noise level. An off-the-shelf classifier (or CLIP) does not work well on very noisy inputs. [Likely]
  • The classifier gradient can be adversarial: a small, unnatural change to \(\mathbf{x}_t\) can raise the classifier score without making the image more like the class. Image gradients of classifiers are famous for this.
  • It needs a classifier for the condition. For free-form text prompts there is no simple classifier \(p(\text{prompt} \mid \mathbf{x})\).
  • Each step needs a backward pass through the classifier, which costs extra time and memory.

Classifier-free guidance

Ho and Salimans asked: can we get the classifier gradient without a classifier? Use Bayes’ rule the other way round. The classifier implied by a conditional and an unconditional diffusion model is \(p(\mathbf{y} \mid \mathbf{x}_t) \propto p(\mathbf{x}_t \mid \mathbf{y}) / p(\mathbf{x}_t)\). Its gradient, written with noise predictions through (227.3), is

\[ \nabla_{\mathbf{x}_t} \log p(\mathbf{y} \mid \mathbf{x}_t) = \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t \mid \mathbf{y}) - \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t) \approx -\frac{\epsilon_\theta(\mathbf{x}_t, t, \mathbf{y}) - \epsilon_\theta(\mathbf{x}_t, t, \varnothing)}{\sqrt{1 - \bar\alpha_t}} \tag{227.6} \]

The difference between the conditional and the unconditional noise prediction is a classifier gradient (up to scale). Now apply guidance with this implicit classifier to the unconditional model, with scale \(w\), and simplify. Write \(\epsilon_c = \epsilon_\theta(\mathbf{x}_t, t, \mathbf{y})\) and \(\epsilon_u = \epsilon_\theta(\mathbf{x}_t, t, \varnothing)\):

\[ \tilde{\epsilon} = \epsilon_u + w\,(\epsilon_c - \epsilon_u) \;=\; w\,\epsilon_c - (w - 1)\,\epsilon_u \tag{227.7} \]
Read the symbols
\(\varnothing\)
the “null” condition: an empty prompt or a special null class id
\(\epsilon_u, \epsilon_c\)
unconditional and conditional noise predictions of the same network
\(w\)
the guidance scale: \(w = 0\) is unconditional, \(w = 1\) is plain conditional, \(w \gt 1\) extrapolates past the conditional prediction
\(\epsilon_c - \epsilon_u\)
the guidance direction: what the condition changes

We start from a point on the line through \(\epsilon_u\) and \(\epsilon_c\), and go \(w\) times the distance in the direction the condition points. It is extrapolation, not interpolation. The sampled distribution, if all scores were exact, would be

\[ \tilde p_w(\mathbf{x}_t \mid \mathbf{y}) \;\propto\; p(\mathbf{x}_t)^{1-w}\, p(\mathbf{x}_t \mid \mathbf{y})^{w} \;\propto\; p(\mathbf{x}_t)\, p(\mathbf{y} \mid \mathbf{x}_t)^{w} \tag{227.11} \]

This is (227.5) with the implicit classifier. So CFG with scale \(w\) on a conditional model is classifier guidance with scale \(w - 1\) on the conditional model, or scale \(w\) on the unconditional one. [Certain] One caveat that matters for the next section: (227.11) is the target at each noise level separately. Running the sampler with the guided noise does not, in general, produce samples of \(\tilde p_w\) at \(t = 0\). [Certain for the toy below; the general statement is Likely]

Training: one network, two jobs

We need \(\epsilon_u\) and \(\epsilon_c\). Training two networks would double the cost. Instead, train one network and, with probability \(p_{\text{uncond}}\), replace the condition with \(\varnothing\):

\[ \mathcal{L}(\theta) = \E_{\mathbf{x}_0, \mathbf{y}, t, \epsilon,\, b \sim \text{Bernoulli}(p_{\text{uncond}})} \Big[ \big\lVert \epsilon - \epsilon_\theta\big(\mathbf{x}_t,\, t,\; b\,\varnothing + (1 - b)\,\mathbf{y}\big) \big\rVert^2 \Big] \tag{227.12} \]

(The “\(b\,\varnothing + (1-b)\,\mathbf{y}\)” only means: use \(\varnothing\) when \(b = 1\), else \(\mathbf{y}\).) Ho and Salimans tried \(p_{\text{uncond}} \in \{0.1, 0.2, 0.5\}\): 0.1 and 0.2 worked about equally well, and 0.5 was worse across the whole quality frontier. [Certain] Stable Diffusion 1.x dropped the text condition 10% of the time. [Certain] For text, \(\varnothing\) is usually the encoding of the empty string “”.

Algorithm 1: training with classifier-free guidance (label dropout)
  1. input: data pairs \((\mathbf{x}_0, \mathbf{y})\), drop rate \(p_{\text{uncond}}\) (often 0.1), schedule \(\bar\alpha_t\)
  2. repeat
  3.   sample \((\mathbf{x}_0, \mathbf{y})\), \(t \sim \text{Uniform}\{1, \dots, T\}\), \(\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\)
  4.   with probability \(p_{\text{uncond}}\): \(\mathbf{y} \leftarrow \varnothing\) // empty prompt or null class
  5.   \(\mathbf{x}_t \leftarrow \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon\)
  6.   gradient step on \(\lVert \epsilon - \epsilon_\theta(\mathbf{x}_t, t, \mathbf{y}) \rVert^2\) // formula (227.12)
  7. until converged
Algorithm 2: DDIM sampling with classifier-free guidance
  1. input: condition \(\mathbf{y}\), scale \(w\), time steps \(t_N \gt \dots \gt t_1\)
  2. \(\mathbf{x} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\)
  3. for \(i = N, \dots, 1\) do
  4.   \([\epsilon_u, \epsilon_c] \leftarrow \epsilon_\theta([\mathbf{x}, \mathbf{x}], t_i, [\varnothing, \mathbf{y}])\) // one batch of 2: two forward passes of work
  5.   \(\tilde{\epsilon} \leftarrow \epsilon_u + w(\epsilon_c - \epsilon_u)\) // formula (227.7)
  6.   \(\hat{\mathbf{x}}_0 \leftarrow (\mathbf{x} - \sqrt{1 - \bar\alpha_{t_i}}\,\tilde{\epsilon}) / \sqrt{\bar\alpha_{t_i}}\)
  7.   \(\mathbf{x} \leftarrow \sqrt{\bar\alpha_{t_{i-1}}}\,\hat{\mathbf{x}}_0 + \sqrt{1 - \bar\alpha_{t_{i-1}}}\,\tilde{\epsilon}\) // DDIM step, \(\bar\alpha_{t_0} = 1\)
  8. end for; return \(\mathbf{x}\)
Worked example: one CFG combination, and the cost

Take a 2-number toy: \(\epsilon_u = (0.2, -0.1)\) and \(\epsilon_c = (0.5, 0.3)\). The guidance direction is \(\epsilon_c - \epsilon_u = (0.3, 0.4)\), with length 0.5. With \(w = 7.5\): \(\tilde{\epsilon} = (0.2 + 7.5 \cdot 0.3,\; -0.1 + 7.5 \cdot 0.4) = (2.45,\; 2.9)\). The guided prediction lies \(6.5 \times 0.5 = 3.25\) beyond the conditional one, far outside the range of values the network ever produced in training. Keep that in mind for the “saturation” problem below.

Cost: 50 DDIM steps with CFG need \(2 \times 50 = 100\) network evaluations, twice the unguided cost. With batching they run as one batch of 2, so latency grows less than 2× on a GPU that is not already full, but the FLOPs double. [Certain for FLOPs]

Watching Algorithm 2 run on 2-D data

To see the formula at work, we need a model whose \(\epsilon_u\) and \(\epsilon_c\) we know exactly. Take 2-D data with three classes A, B and C. Each class is a round Gaussian \(\mathcal{N}(\mu_k, s^2 \mathbf{I})\) with \(s = 0.55\), and the classes are equally likely. The noisy data \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\epsilon\) of one class is again Gaussian, with mean \(\sqrt{\bar\alpha_t}\,\mu_k\) and variance \(v_t = \bar\alpha_t s^2 + 1 - \bar\alpha_t\) per axis. So the best possible noise prediction (the one a perfect network would learn) has a closed form:

\[ \epsilon_k(\mathbf{x}, t) = \frac{\sqrt{1-\bar\alpha_t}}{v_t}\,\big(\mathbf{x} - \sqrt{\bar\alpha_t}\,\mu_k\big), \qquad v_t = \bar\alpha_t s^2 + 1 - \bar\alpha_t \tag{227.8} \]

The conditional prediction for class \(\mathbf{y} = k\) is \(\epsilon_c = \epsilon_k\). The unconditional prediction does not know the class, so it averages the three class predictions, weighted by how likely each class is given the noisy point:

\[ \epsilon_u(\mathbf{x}, t) = \sum_{k} r_k(\mathbf{x})\, \epsilon_k(\mathbf{x}, t), \qquad r_k(\mathbf{x}) = \softmax_k\!\left( -\frac{\lVert \mathbf{x} - \sqrt{\bar\alpha_t}\,\mu_k \rVert^2}{2 v_t} \right) \tag{227.9} \]
Read the symbols
\(\mu_k\)
the center of class \(k\): A at \((-1.6, -0.9)\), B at \((1.6, -0.9)\), C at \((0, 1.8)\)
\(v_t\)
the variance of the noisy class data at step \(t\): almost 1 at the start (pure noise), \(s^2\) at the end
\(r_k(\mathbf{x})\)
the posterior probability of class \(k\) for the noisy point: the implicit classifier of (227.6)

Because \(\hat{\mathbf{x}}_0 = (\mathbf{x} - \sqrt{1-\bar\alpha_t}\,\epsilon)/\sqrt{\bar\alpha_t}\) is a straight-line (affine) function of \(\epsilon\), the CFG mix carries over to the clean-image guesses with the same weights:

\[ \hat{\mathbf{x}}_0^{\text{cfg}} = \hat{\mathbf{x}}_0^{u} + w\,\big(\hat{\mathbf{x}}_0^{c} - \hat{\mathbf{x}}_0^{u}\big) \tag{227.10} \]

This is what Figure 2 draws. For the tracked point, the blue dot is where the unconditional model thinks the clean point is, the green dot is where the conditional model thinks it is, and the orange dot is the guided guess, \(w\) times as far along the same line. With \(w \gt 1\) the orange dot lies beyond the green one, on the side away from the other classes.

Condition \(\mathbf{y}\):
samples\(\hat{\mathbf{x}}_0\) from \(\epsilon_u\)\(\hat{\mathbf{x}}_0\) from \(\epsilon_c\)\(\hat{\mathbf{x}}_0\) from \(\tilde\epsilon\)
  1. \(\mathbf{x} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) // 240 points
  2. for \(i = 20, \dots, 1\) do // \(t_i\) from 999 to 0
  3. \(\epsilon_u, \epsilon_c \leftarrow \epsilon(\mathbf{x}, t_i, \varnothing),\ \epsilon(\mathbf{x}, t_i, \mathbf{y})\) // (227.9), (227.8)
  4. \(\tilde\epsilon \leftarrow \epsilon_u + w(\epsilon_c - \epsilon_u)\) // (227.7)
  5. \(\hat{\mathbf{x}}_0 \leftarrow (\mathbf{x} - \sqrt{1-\bar\alpha_{t_i}}\,\tilde\epsilon)/\sqrt{\bar\alpha_{t_i}}\)
  6. \(\mathbf{x} \leftarrow \sqrt{\bar\alpha_{t_{i-1}}}\,\hat{\mathbf{x}}_0 + \sqrt{1-\bar\alpha_{t_{i-1}}}\,\tilde\epsilon\)
  7. return \(\mathbf{x}\)
final spread (std per axis)mean distance to the class centertrue class spread 0.55 (dashed)
Figure 2. Algorithm 2 computed in your browser. 240 starting points (seeded) run 20 DDIM steps of the linear schedule (\(\beta\) from \(10^{-4}\) to 0.02, \(T = 1000\)), with the exact noise predictions (227.8) and (227.9). Circles: the three classes at \(t = 0\) (radius \(2s\)); dashed circles: the same classes at the current noise level. The trail is the path of the tracked point; click any point to track it. Bottom: the final spread and distance to the class center for every \(w\) from 0 to 10 (same starting points), with the current \(w\) marked.
How to read Figure 2

Press Play. Each DDIM step runs lines 3 to 6; the panel on the right shows the formula of the current line with the real numbers of the tracked point. Early on (\(\bar\alpha_t\) near 0) all three guesses \(\hat{\mathbf{x}}_0\) sit near the middle, because the noisy point says little. As the noise drops, they separate and the points fall into place. The tracked point starts at \((2.76, 1.13)\), near class B. Set \(w = 0\) (unconditional): it ends in class B, at \((2.60, 0.03)\). With \(w = 1\) it ends in class C at \((1.32, 2.34)\); with \(w = 3\) at \((1.12, 2.93)\); with \(w = 7.5\) at \((1.00, 3.60)\), above the class C circle. Now drag \(w\) while paused: the whole run is recomputed and you see the same step with the new scale. In the chart at the bottom, the spread keeps falling as \(w\) grows, while the distance to the class center is smallest near \(w = 1.5\) and then grows: the samples become narrow and pushed away from the other classes, “more C than real C data”.

Worked example: one guided DDIM step from Figure 2

Class C, \(w = 3\), the tracked point at step 11 of 20 (\(t = 473\), \(\bar\alpha_t = 0.1015\), so \(\sqrt{\bar\alpha_t} = 0.3186\) and \(\sqrt{1-\bar\alpha_t} = 0.9479\)). The point is \(\mathbf{x} = (2.409, 2.542)\). The exact predictions are \(\epsilon_u = (2.334, 2.250)\) and \(\epsilon_c = (2.458, 2.009)\), so the guidance direction is \(\epsilon_c - \epsilon_u = (0.124, -0.241)\).

Line 4: \(\tilde\epsilon = (2.334, 2.250) + 3 \cdot (0.124, -0.241) = (2.706, 1.526)\). Line 5: \(\hat{\mathbf{x}}_0 = \big((2.409, 2.542) - 0.9479 \cdot (2.706, 1.526)\big) / 0.3186 = (-0.488, 3.439)\). The guided guess is above class C’s center \((0, 1.8)\): guidance removes less noise along the second axis, so the point is pushed up, away from classes A and B below.

The final samples (all 240 points, class C) in numbers: with \(w = 0\), only 27.1% land in class C’s region (the region where C’s center is the nearest). With \(w = 1\), 99.6% do, with a spread (standard deviation per axis) of 0.48. With \(w = 3\): 100%, spread 0.33, mean at \(y = 2.47\). With \(w = 7.5\): 100%, spread 0.28, mean at \(y = 3.22\), far above the true center 1.8. The true class spread is 0.55; with 20 steps the sampler gives a bit less (0.48 at \(w = 1\); 0.54 with 200 steps). [Certain for this toy]

Negative prompts

Nothing forces \(\epsilon_u\) to use the empty prompt. If you replace it with the prediction for a negative prompt \(\mathbf{y}^-\) (“blurry, low quality, extra fingers”), formula (227.7) becomes \(\epsilon_{\mathbf{y}^-} + w(\epsilon_{\mathbf{y}} - \epsilon_{\mathbf{y}^-})\): the sampler moves toward the prompt and away from the negative prompt. In diffusers this is the negative_prompt argument; its embedding simply takes the place of the empty-prompt embedding in the batch. [Certain]

Why high guidance gives sharp but less varied images

Let us see the effect exactly on a problem small enough to compute. One-dimensional data, two classes with equal probability: class “+” is \(\mathcal{N}(+1, 0.5^2)\), class “−” is \(\mathcal{N}(-1, 0.5^2)\). Because everything is Gaussian, the true scores of the noisy distributions are known in closed form at every noise level. No network, no training error. We condition on class “+” and run the exact probability-flow ODE (the deterministic sampler of session 226) with the guided score \(s_u + w(s_c - s_u)\) on 400 evenly spaced starting points.

data \(p(x)\), both classesclass “+” \(p(x \mid +)\)guided samples (histogram)tilted target (dashed, at the end)plain conditional \(w = 1\) samples (outline)
Figure 3. Exact classifier-free guidance on a 1-D, two-class problem. Orange bars: where 400 deterministic samples end up when the probability-flow ODE (120 Heun steps, \(\sigma\) from 80 down to 0) uses the guided score. The shaded region \(x \lt 0\) is where a Bayes classifier says “class −”. “Play the sampler” replays the last 90 of the 122 noise levels (\(\sigma\) from 4.7 down to 0): the bars are the samples at the current \(\sigma\), the curves are the noisy densities at that \(\sigma\), and the dark outline shows the plain conditional (\(w = 1\)) samples at the same \(\sigma\).
How to read Figure 3

At \(w = 0\) the samples cover both classes (unconditional). At \(w = 1\) they match the blue class-“+” curve: plain conditional sampling, and about 2% of samples still fall on the “−” side, exactly like real data of class “+”. Increase \(w\): the histogram moves away from the other class and gets narrower. Nothing ends up on the wrong side, but the spread (diversity) shrinks. Compare the orange histogram with the green dashed curve: the guided sampler pushes much farther than the “tilted” target (227.11) evaluated at \(t = 0\), because guidance acts at every noise level and the effects add up along the path. Press “Play the sampler” (with \(w = 3\)) to watch this happen. At \(\sigma = 4.7\) the guided cloud is already shifted: its mean is 2.74, against 0.94 for the plain conditional (\(w = 1\)) samples, because at high noise the guidance term moves the whole cloud. As \(\sigma\) falls, the class score pulls the samples back, but not all the way: the mean ends at 1.51 versus 0.99. The narrowing comes later: the guided spread is 0.96 times the plain one at \(\sigma = 4.7\), 0.89 times at \(\sigma = 2.8\), and 0.63 times from \(\sigma \approx 0.6\) on (0.31 versus 0.50 at the end).

Worked example: the numbers behind Figure 3

The implicit classifier. For this data, \(p(+ \mid x) = 1/(1 + e^{-8x})\) at \(t = 0\) (the factor is \(2 \cdot 1 / 0.5^2 = 8\)). At \(x = 0.3\): \(p(+ \mid x) = 0.917\), the class score is \(s_c = -(0.3 - 1)/0.25 = 2.8\), and the unconditional score is \(s_u = 2.135\). The difference \(s_c - s_u = 0.665\) equals \(\frac{d}{dx}\log p(+ \mid x) = 8\,(1 - 0.917)\), as (227.6) says. With \(w = 3\), the guided score is \(2.135 + 3 \cdot 0.665 = 4.131\): the point is pushed toward larger \(x\) much harder than the plain conditional score 2.8.

The samples. Computed with the ODE (and checked in Python with the same algorithm):

\(w\)meanstd (diversity)share classified “+”
00.001.1250%
10.990.5097.8%
31.510.31100%
7.52.060.25100%

From \(w = 1\) to \(w = 7.5\), the spread halves and the mean moves to twice the true class center. The samples are “more class +” than the real class + data ever is. In images, that is the over-saturated, over-sharpened, “too perfect” look of very high CFG. The tilted target (227.11) at \(t = 0\) with \(w = 7.5\) would have mean 1.08 and std 0.44: the real sampler goes much further. [Certain for this toy]

The published trade-off curves

Ho and Salimans swept the guidance strength on ImageNet 64 × 64 and measured FID and IS. Figure 4 plots their Table 1. Their best FID came from small guidance (\(w' = 0.1\) to \(0.3\)), and the best IS from strong guidance (\(w' \ge 4\)). [Certain] Note how small the FID-optimal scale is, compared with the 7.5 used for text-to-image. FID rewards diversity; people judging single images prefer more guidance. [Likely]

Figure 4. FID (lower is better) against Inception Score (higher is better) for classifier-free guidance on ImageNet 64 × 64, from Table 1 of Ho and Salimans (2022). Each dot is one guidance strength \(w' \in \{0, 0.1, \dots, 1, 2, 3, 4\}\); the three curves are three label-dropout rates.
How to read Figure 4

Moving right along a curve means more guidance. IS keeps rising: each image is more clearly one class. FID first drops (better images), reaches its minimum at \(w' = 0.1\) for \(p_{\text{uncond}} = 0.1\), then rises steeply as diversity collapses. Switch to \(p_{\text{uncond}} = 0.5\): the whole curve is worse, because half the training went to the unconditional task.

What goes wrong at very high guidance, and the fixes

  • Saturation and over-exposure. As the worked example showed, \(\tilde{\epsilon}\) leaves the range the network was trained on, so the implied \(\hat{\mathbf{x}}_0\) has values far outside the valid pixel range. Imagen introduced dynamic thresholding: at each step, clip \(\hat{\mathbf{x}}_0\) to a percentile of its absolute values and rescale. [Certain]
  • Guidance rescale. Lin et al. rescale the guided prediction so its standard deviation matches the conditional one, then mix:
    \[ \mathbf{x}_{\text{res}} = \mathbf{x}_{\text{cfg}} \cdot \frac{\operatorname{std}(\mathbf{x}_{c})}{\operatorname{std}(\mathbf{x}_{\text{cfg}})}, \qquad \mathbf{x}_{\text{final}} = \phi\, \mathbf{x}_{\text{res}} + (1 - \phi)\, \mathbf{x}_{\text{cfg}} \tag{227.13} \]
    They report that \(w = 7.5\), \(\phi = 0.7\) works well. [Certain] In diffusers this is guidance_rescale, applied to the noise prediction.
  • Error amplification. A learned \(\epsilon_c - \epsilon_u\) has errors, and \(w\) multiplies them. In the PyTorch toy below, a small trained network gives the right answer at \(w = 3\) but at \(w = 7.5\) its samples are wider (std 1.53) instead of narrower (exact: 0.25 with the same sampler). [Certain, our run]
  • Guidance only where it helps. Kynkäänniemi et al. apply guidance only in a middle range of noise levels: at the highest noise it mostly destroys diversity, at the lowest it does little. [Likely]
  • Removing the second pass. Distilled models learn to output the guided prediction directly, with \(w\) as an input, so one forward pass per step is enough. Some open “guidance-distilled” models work this way. [Likely]

ControlNet: spatial control for a frozen model

Text says what to draw but is weak at saying where. “A dancer with her left arm raised” rarely gets the exact pose. ControlNet adds a spatial condition: an image of the same size that fixes the layout. Examples: Canny edges, a depth map, a normal map, straight lines (M-LSD), soft edges (HED), a segmentation map, a human pose skeleton (OpenPose), a scribble. [Certain]

The challenge: Stable Diffusion was trained on billions of images; a pose dataset may have only thousands. Fine-tuning the whole model on it would cause overfitting and forgetting. ControlNet’s answer has three parts.

  1. Lock the original network. Its weights \(\Theta\) never change.
  2. Make a trainable copy \(\Theta_c\) of the encoder blocks, initialized with the same weights. The copy is a strong, pretrained feature extractor from the first step.
  3. Connect them with zero convolutions \(\mathcal{Z}\): \(1 \times 1\) convolutions whose weight and bias start at exactly zero.
\[ \mathbf{y}_c = \mathcal{F}(\mathbf{x}; \Theta) + \mathcal{Z}\Big( \mathcal{F}\big(\mathbf{x} + \mathcal{Z}(\mathbf{c}; \Theta_{z1});\; \Theta_c \big);\; \Theta_{z2} \Big) \tag{227.14} \]
Read the symbols
\(\mathcal{F}(\cdot; \Theta)\)
a pretrained block with locked weights; \(\mathbf{x}\) is its input feature map
\(\mathcal{F}(\cdot; \Theta_c)\)
the trainable copy of the same block
\(\mathbf{c}\)
the condition features (edge map, pose, depth), encoded to the latent size
\(\mathcal{Z}(\cdot; \Theta_{z})\)
a \(1 \times 1\) convolution with weight and bias initialized to 0
\(\mathbf{y}_c\)
the output: the original output plus a learned correction

At the first training step both \(\mathcal{Z}\) terms are zero, so \(\mathbf{y}_c = \mathcal{F}(\mathbf{x}; \Theta)\) exactly. The model starts as the unchanged pretrained model and can only improve from there. No random noise is injected into the features at the start of training. [Certain]

Why zero does not mean “stuck”

If a layer’s weights are zero, can it still learn? Yes, if its input is not zero. For a \(1 \times 1\) convolution \(\mathbf{z} = W\mathbf{h} + \mathbf{b}\) at one position,

\[ \frac{\partial \mathcal{L}}{\partial W} = \frac{\partial \mathcal{L}}{\partial \mathbf{z}}\, \mathbf{h}^\top, \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{b}} = \frac{\partial \mathcal{L}}{\partial \mathbf{z}}, \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{h}} = W^\top \frac{\partial \mathcal{L}}{\partial \mathbf{z}} \tag{227.15} \]

With \(W = 0\), the first two are non-zero (the copy’s features \(\mathbf{h}\) are not zero), but the third is zero. So at step 1 only the output zero convolution moves. From step 2 on, \(W \ne 0\), gradients flow into the trainable copy and into the input zero convolution. Figure 5 shows this on a toy you can follow by hand.

\(y = 1.5x + v\,\theta_c\,(x + u\,c) + b\); bold boxes have a non-zero gradient at this step

Figure 5. A one-number ControlNet. Locked block \(F(x) = 1.5x\); trainable copy \(F_c(x) = \theta_c x\) with \(\theta_c = 1.5\) at the start; zero convolutions \(z_1 = u\,c\) and \(z_2 = v\,h + b\) with \(u = v = b = 0\). The output is \(y = 1.5x + v\,\theta_c (x + u c) + b\). Four training pairs \((x, c)\) with target \(1.5x + 0.5c\); plain gradient descent, learning rate 0.03, 1,500 steps.
How to read Figure 5

Step 0: the output is exactly the locked model’s output, and the loss equals the base model’s loss, 0.375. Only \(v\) and \(b\) (the output zero convolution) have non-zero gradients; \(u\) and \(\theta_c\) have gradient exactly 0, as (227.15) predicts. From step 1 on, all parameters move. The last step compares with a random (non-zero) initialization: the loss at step 0 is 2.09, much worse than the pretrained model, which is the damage zero convolutions avoid.

ControlNet on Stable Diffusion

The details from the paper [Certain]:

  • The trainable copy covers the 12 encoder blocks and the middle block of the SD U-Net (4 resolutions, 64, 32, 16 and 8, three blocks each). Their outputs, through zero convolutions, are added to the 12 skip connections and the middle block of the locked U-Net.
  • The condition image (512 × 512) is turned into a 64 × 64 feature map by a tiny encoder: four convolutions with 4 × 4 kernels and stride 2, with 16, 32, 64 and 128 channels.
  • The locked part needs no gradients, so training costs only about 23% more GPU memory and 34% more time per iteration than fine-tuning SD alone (one A100 40 GB).
  • 50% of text prompts are replaced by empty strings during training, so the ControlNet learns to read the meaning of the condition image itself.
  • Sudden convergence: the model does not learn the condition gradually; it suddenly starts to follow it, usually in fewer than 10K steps.
  • Training works with small (under 50k) and large (over 1M) datasets. Several ControlNets (for example pose and depth) can be combined by adding their outputs.
Algorithm 3: one denoising step of Stable Diffusion with a ControlNet and CFG
  1. input: latent \(\mathbf{x}_t\), step \(t\), text embeddings \(\tau(\mathbf{y}), \tau(\varnothing)\), condition image \(\mathbf{c}_{\text{img}}\), CFG scale \(w\), control strength \(\lambda\)
  2. \(\mathbf{c} \leftarrow E(\mathbf{c}_{\text{img}})\) // tiny conv encoder, \(512 \to 64\), computed once
  3. for each text input \(e \in \{\tau(\varnothing), \tau(\mathbf{y})\}\) do // one batch of 2
  4.   \(\{\mathbf{r}_1, \dots, \mathbf{r}_{12}, \mathbf{r}_{\text{mid}}\} \leftarrow \text{ControlNet}(\mathbf{x}_t, t, e, \mathbf{c})\) // trainable copy + zero convs, (227.14)
  5.   \(\epsilon_e \leftarrow \text{U-Net}(\mathbf{x}_t, t, e;\ \text{skip}_k \mathrel{+}= \lambda\,\mathbf{r}_k,\ \text{mid} \mathrel{+}= \lambda\,\mathbf{r}_{\text{mid}})\)
  6. end for
  7. \(\tilde{\epsilon} \leftarrow \epsilon_u + w(\epsilon_c - \epsilon_u)\); take the sampler step as in Algorithm 2

A subtle point: should the control be added to both the conditional and the unconditional branch of CFG? The paper reports that adding it to both removes the guidance when there is no prompt, while adding it only to \(\epsilon_c\) makes guidance too strong; their fix (“CFG resolution weighting”) adds it to \(\epsilon_c\) with a weight \(64/h_i\) that depends on the block resolution \(h_i\). [Certain] Common libraries add the control to both branches by default. [Likely]

Lighter alternatives exist. T2I-Adapter trains a small separate network whose features are added to the U-Net encoder, with far fewer parameters than a full encoder copy. [Likely] The same zero-initialization trick appears in many places: the SD inpainting model initialized its 5 new input channels with zero weights [Certain], and LoRA starts one of its two matrices at zero (session 57).

Code: classifier-free guidance and a zero-convolution block in PyTorch

The first program trains a tiny class-conditional diffusion model on the 1-D, two-class data of Figure 3, with label dropout (Algorithm 1), then samples with CFG and DDIM (Algorithm 2). It runs in about 20 seconds on a laptop CPU and is deterministic.

import math, torch, torch.nn as nn
torch.manual_seed(0)

# 1) toy data: two classes on a line, each N(+-1, 0.5^2)
def data(n):
    c = torch.randint(0, 2, (n,))
    x = (2 * c - 1).float() * 1.0 + 0.5 * torch.randn(n)
    return x[:, None], c

T = 1000
betas = torch.linspace(1e-4, 0.02, T)
abar = torch.cumprod(1 - betas, 0)              # alpha-bar_t

class Eps(nn.Module):                           # eps_theta(x_t, t, c); label 2 = "no label"
    def __init__(s, h=256):
        super().__init__()
        s.emb = nn.Embedding(3, 16)
        s.net = nn.Sequential(nn.Linear(1 + 16 + 16, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(),
                              nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 1))
    def forward(s, x, t, c):
        f = torch.exp(-math.log(1e4) * torch.arange(8) / 8) * t[:, None].float()
        return s.net(torch.cat([x, f.sin(), f.cos(), s.emb(c)], 1))

model = Eps()
opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, 6000)
P_UNCOND = 0.1
for step in range(6000):                         # 2) training with label dropout, (227.12)
    x0, c = data(512)
    c = torch.where(torch.rand(512) < P_UNCOND, torch.full_like(c, 2), c)
    t = torch.randint(0, T, (512,))
    eps = torch.randn_like(x0)
    xt = abar[t, None].sqrt() * x0 + (1 - abar[t, None]).sqrt() * eps
    loss = ((model(xt, t, c) - eps) ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step(); sched.step()

@torch.no_grad()
def sample(w, label=1, n=2000, steps=50):        # 3) DDIM sampling with CFG
    g = torch.Generator().manual_seed(1)
    x = torch.randn(n, 1, generator=g)
    ts = torch.linspace(T - 1, 0, steps).long()
    for i, t in enumerate(ts):
        tt = t.repeat(n)
        x2, t2 = torch.cat([x, x]), torch.cat([tt, tt])          # one batch, two halves
        c2 = torch.cat([torch.full((n,), 2), torch.full((n,), label)])
        e_u, e_c = model(x2, t2, c2).chunk(2)
        e = e_u + w * (e_c - e_u)                                # formula (227.7)
        x0 = (x - (1 - abar[t]).sqrt() * e) / abar[t].sqrt()
        ab_prev = abar[ts[i + 1]] if i + 1 < steps else torch.tensor(1.0)
        x = ab_prev.sqrt() * x0 + (1 - ab_prev).sqrt() * e
    return x

for w in [0.0, 1.0, 3.0, 7.5]:
    x = sample(w)
    print(f"w={w:4}: mean {x.mean():.2f}  std {x.std():.2f}  share x>0 {(x > 0).float().mean():.3f}")

Our run (PyTorch 2.15, CPU) printed:

w= 0.0: mean 0.03  std 1.11  share x>0 0.517
w= 1.0: mean 0.99  std 0.49  share x>0 0.980
w= 3.0: mean 1.47  std 0.36  share x>0 1.000
w= 7.5: mean 2.70  std 1.53  share x>0 1.000

Compare with the exact values from the same DDIM sampler with the true noise predictions: 0.02 / 1.09, 1.00 / 0.47, 1.51 / 0.29, 2.06 / 0.23 (mean / std). Up to \(w = 3\) the small network is close. At \(w = 7.5\), its small errors in \(\epsilon_c - \epsilon_u\) are multiplied by 7.5, and the samples spread out instead of tightening. Exact numbers can change slightly with the PyTorch version or hardware.

The second program builds a ControlNet-style block, formula (227.14), and checks the two facts from Figure 5: the output equals the locked block at initialization, and only the output zero convolution gets a gradient at step 0.

import copy, torch, torch.nn as nn
torch.manual_seed(0)

def zero_conv(ch):
    z = nn.Conv2d(ch, ch, 1); nn.init.zeros_(z.weight); nn.init.zeros_(z.bias); return z

class ControlledBlock(nn.Module):
    """y_c = F(x) + Z2(F_copy(x + Z1(c)))  -- formula (227.14)"""
    def __init__(s, block, ch):
        super().__init__()
        s.locked = block.requires_grad_(False)              # frozen original weights
        s.copy = copy.deepcopy(block).requires_grad_(True)  # trainable copy, same start
        s.z1, s.z2 = zero_conv(ch), zero_conv(ch)
    def forward(s, x, c):
        return s.locked(x) + s.z2(s.copy(x + s.z1(c)))

block = nn.Sequential(nn.Conv2d(8, 8, 3, padding=1), nn.SiLU(), nn.Conv2d(8, 8, 3, padding=1))
cb = ControlledBlock(block, 8)
x, c = torch.randn(2, 8, 16, 16), torch.randn(2, 8, 16, 16)
y = cb(x, c)
print((y - block(x)).abs().max().item())            # 0.0: the pretrained model is unchanged
y.pow(2).mean().backward()
print(cb.z2.weight.grad.abs().sum().item() > 0,     # True: the output zero conv learns first
      cb.z1.weight.grad.abs().sum().item(),          # 0.0 at step 0
      cb.copy[0].weight.grad.abs().sum().item())     # 0.0 at step 0

It prints 0.0, then True 0.0 0.0.

Where this is used

System or taskConditioning and guidanceNotes
Stable Diffusion 1.x / 2.x, SDXLCross-attention to frozen CLIP text encoders; CFG with the empty prompt as \(\varnothing\); diffusers default scale 7.5 with 50 steps10% caption dropout in SD 1.x training [Certain]
Class-conditional ImageNet models (ADM, DiT)Class embedding added to the time embedding; classifier guidance (ADM-G) or CFGThe standard benchmark for the FID / IS trade-off
Controlled image generation and editingControlNet with edges, depth, pose, segmentation; several ControlNets added togetherLayout from the condition, content from the prompt
Text-to-video generationCross- or joint attention to text, CFG on every frame’s latent; often a lower scale than for images [Likely]Too much guidance shows as flicker and over-saturated colors over time [Speculation]
Pose- or depth-controlled video (motion transfer)ControlNet-style branches fed with per-frame pose skeletons or depth maps [Likely]Pose extracted from a source video drives a new character
Synthetic training data for video tasks (action recognition, segmentation, key-moment detection)Pose or segmentation-map conditioning to create labeled frames; guidance scale tuned for diversity, not beautyHigh CFG reduces diversity, which hurts a training set [Likely]
Autoregressive image generators and LLMsCFG on logits: \(\ell_u + w(\ell_c - \ell_u)\) between conditional and unconditional next-token logits [Likely]The same idea outside diffusion (session 79)

Common mistakes

  • Mixing the two scale conventions. A paper’s \(w' = 3\) is a diffusers guidance_scale = 4. Always check whether 1 means “plain conditional” or 0 does.
  • Forgetting label dropout in training. Without it, \(\epsilon_\theta(\cdot, \varnothing)\) was never trained, and CFG combines a good prediction with garbage. You cannot add CFG to a model after training without fine-tuning.
  • Using a different null embedding at sampling time than in training (for example zeros instead of the empty-prompt encoding). The unconditional branch must see exactly what it was trained with.
  • Swapping \(\epsilon_u\) and \(\epsilon_c\) after chunk(2). The order in the batch must match the order you unpack; a swap guides away from the prompt.
  • Turning the scale up to fix a bad prompt. Past a point, higher \(w\) only gives saturated, same-looking images and amplifies model errors. Use rescale, thresholding or a better prompt.
  • Expecting \(w\) to transfer between samplers, step counts and models. The best scale depends on all three; re-tune it.
  • Initializing the ControlNet connection randomly or fine-tuning the locked model too. Both damage the pretrained model at the start of training; keep the zero convolutions and the lock.
  • A condition image that does not match the training preprocessing (a Canny map with different thresholds, a pose in a different keypoint format). ControlNets are trained on one specific detector’s output.

Interview questions

How does a text prompt enter a Stable Diffusion U-Net?basic
A frozen text encoder (CLIP ViT-L/14 in SD 1.x) turns the prompt into 77 token vectors of size 768. Inside the U-Net, each transformer block runs self-attention over the image tokens, then cross-attention where the queries come from the image features and the keys and values come from the text tokens, then a feed-forward layer. Each spatial position pulls in a weighted mix of the word meanings. The time step enters separately through an embedding added in the residual blocks.
Write the classifier-free guidance formula and explain each term.basic
\(\tilde{\epsilon} = \epsilon_u + w(\epsilon_c - \epsilon_u)\), where \(\epsilon_c\) is the noise prediction with the prompt, \(\epsilon_u\) without it (empty prompt), and \(w\) the scale. \(w = 1\) is plain conditional sampling; \(w \gt 1\) extrapolates in the direction the prompt changes the prediction. Both predictions come from the same network, usually computed as one batch of 2. In Ho and Salimans’ notation it is \((1 + w')\epsilon_c - w'\epsilon_u\) with \(w = 1 + w'\).
Derive classifier guidance from Bayes’ rule.core
\(\log p(\mathbf{x}_t \mid \mathbf{y}) = \log p(\mathbf{x}_t) + \log p(\mathbf{y} \mid \mathbf{x}_t) - \log p(\mathbf{y})\); the last term does not depend on \(\mathbf{x}_t\), so the scores add: \(\nabla \log p(\mathbf{x}_t \mid \mathbf{y}) = \nabla \log p(\mathbf{x}_t) + \nabla \log p(\mathbf{y} \mid \mathbf{x}_t)\). With \(\nabla \log p(\mathbf{x}_t) = -\epsilon_\theta/\sqrt{1 - \bar\alpha_t}\), the guided noise prediction is \(\epsilon_\theta - s\sqrt{1 - \bar\alpha_t}\,\nabla \log p_\phi(\mathbf{y} \mid \mathbf{x}_t)\). A scale \(s \gt 1\) corresponds to sampling \(p(\mathbf{x})\,p(\mathbf{y} \mid \mathbf{x})^s\), a sharper version. The classifier must be trained on noisy images.
Why is it called “classifier-free”? Where is the classifier hidden?core
By Bayes, \(p(\mathbf{y} \mid \mathbf{x}_t) \propto p(\mathbf{x}_t \mid \mathbf{y})/p(\mathbf{x}_t)\), so the classifier gradient equals the conditional score minus the unconditional score, i.e. \(-(\epsilon_c - \epsilon_u)/\sqrt{1-\bar\alpha_t}\). Plugging this implicit classifier into classifier guidance gives the CFG formula. No separate classifier is trained; the generative model plays both roles. Note that with learned networks \(\epsilon_c - \epsilon_u\) is not exactly the gradient of any classifier, which also means it cannot be “attacked” the way a real classifier gradient can.
How do you train a model so that CFG works, and what drop rate do you use?core
One network, trained with the usual noise-prediction loss, but with probability \(p_{\text{uncond}}\) the condition is replaced by a null token (empty prompt or null class). Then the same weights give both \(\epsilon_c\) and \(\epsilon_u\). Ho and Salimans found 0.1 and 0.2 about equal and 0.5 worse; Stable Diffusion 1.x used 10%. Too high a rate wastes capacity on the unconditional task; zero makes \(\epsilon_u\) undefined.
Why does high guidance give sharp but less diverse images?core
Guidance samples (approximately) \(p(\mathbf{x})\,p(\mathbf{y} \mid \mathbf{x})^w\). Raising the class probability to a power \(w \gt 1\) concentrates mass where the class is most certain: far from other classes, at the most typical modes. Fewer modes and less spread means lower diversity (recall), while each sample is more clearly the class (precision, IS). On ImageNet 256, guidance scale 10 on a conditional model moved precision from 0.69 to 0.88 and recall from 0.63 to 0.32. In a 1-D two-Gaussian toy, going from \(w = 1\) to \(7.5\) halves the sample standard deviation and doubles the distance from the decision boundary.
What is the cost of CFG at inference, and how can you reduce it?core
Two network evaluations per step (conditional and unconditional), so twice the FLOPs: 100 evaluations for 50 steps. They are usually batched together, so latency grows less than 2× when the GPU has spare capacity. Reductions: guidance distillation (a student model takes \(w\) as input and outputs the guided prediction in one pass), applying guidance only in part of the noise range, caching or skipping the unconditional pass on some steps, and fewer sampling steps (consistency or distilled models). [Likely for the last three]
Images at guidance scale 15 look over-saturated. Why, and what can you do?advanced
The guided noise is an extrapolation far beyond anything the network produced in training, so \(\hat{\mathbf{x}}_0 = (\mathbf{x}_t - \sqrt{1-\bar\alpha_t}\,\tilde{\epsilon})/\sqrt{\bar\alpha_t}\) gets values outside the valid range, and errors in \(\epsilon_c - \epsilon_u\) are multiplied by \(w\). Fixes: lower the scale; dynamic thresholding (Imagen: clip \(\hat{\mathbf{x}}_0\) at a percentile and rescale); guidance rescale (Lin et al.: match the standard deviation of the guided prediction to the conditional one, mix with \(\phi = 0.7\)); apply guidance only in a middle range of noise levels.
Explain ControlNet and why its convolutions start at zero.core
ControlNet locks the pretrained U-Net and makes a trainable copy of its 12 encoder blocks and middle block. The copy receives the latent plus the encoded condition image (edges, pose, depth), and its outputs are added to the locked U-Net’s skip connections and middle block through \(1 \times 1\) convolutions initialized with zero weight and bias. At the start, those convolutions output zero, so the model is exactly the pretrained model: no random noise disturbs it, and training begins from a good point. The zero layers still learn because their gradient \(\partial L/\partial W = (\partial L/\partial z)\,\mathbf{h}^\top\) depends on their non-zero input; after one step gradients reach the trainable copy.
Why make a copy of the encoder instead of training a small new network for the condition?advanced
The copy starts with the pretrained weights, so it already understands images at all U-Net scales and produces features in exactly the format the locked decoder expects. That gives a deep, strong backbone that learns from small datasets (the paper reports success under 50k images) without overfitting, and the locked original keeps the model’s quality. The cost is about as many trainable parameters as the encoder and, in the paper’s setup, about 23% more memory and 34% more time per step than plain fine-tuning. Lighter adapters (T2I-Adapter) trade some control quality for size.
Does sampling with CFG at scale \(w\) produce samples from \(p(\mathbf{x})^{1-w} p(\mathbf{x} \mid \mathbf{y})^w\)?advanced
Not in general. That tilted distribution is the target at each noise level separately, but the noisy versions of the tilted clean distribution are not the tilted noisy distributions, so the guided score is not the score of any single diffusion path. On a 1-D two-Gaussian problem with exact scores, \(w = 7.5\) gives samples with mean 2.06 and std 0.25, while the tilted clean distribution has mean 1.08 and std 0.44: the sampler goes much further. CFG is best seen as a useful heuristic with a Bayes-rule motivation.

Exercises

  1. A paper reports its best FID at \(w' = 0.3\) (Ho and Salimans convention). What guidance_scale is that in diffusers? And what is the default 7.5 in the paper’s convention?
  2. Compute \(\tilde{\epsilon}\) for \(\epsilon_u = (1, 0)\), \(\epsilon_c = (0, 1)\) and \(w = 2\). What happens at \(w = 0.5\)?
  3. How many U-Net evaluations does one 30-step image need with CFG? With CFG and one ControlNet (count a ControlNet pass as one evaluation)?
  4. Assume \(p(x \mid c) = \mathcal{N}(1, 0.5^2)\) and \(p(x) = \mathcal{N}(0, 1.25)\) (a Gaussian stand-in for the mixture). Find the mean and standard deviation of \(p(x)^{1-w}p(x \mid c)^w\) for \(w = 3\). Hint: add precisions.
  5. Show that classifier guidance with scale \(s\) on a conditional model equals classifier guidance with scale \(s + 1\) on the unconditional model (exact scores).
  6. In Figure 5, explain why \(\partial L/\partial u = 0\) at step 0 but \(\partial L/\partial v \ne 0\). What would happen if \(\theta_c\) were also initialized to 0?
Answerscheck
1. \(w = 1 + w' = 1.3\). The default 7.5 is \(w' = 6.5\). 2. \(\epsilon_c - \epsilon_u = (-1, 1)\), so \(\tilde{\epsilon} = (1, 0) + 2(-1, 1) = (-1, 2)\). At \(w = 0.5\): \((0.5, 0.5)\), the midpoint, a weaker-than-conditional mix (interpolation). 3. \(2 \times 30 = 60\) U-Net evaluations; with a ControlNet run on both CFG branches, another 60 ControlNet passes, 120 in total. 4. Precision \(= w/0.25 + (1 - w)/1.25 = 12 - 1.6 = 10.4\), variance \(1/10.4 = 0.096\), std \(0.31\); mean \(= (w \cdot 1/0.25)/10.4 = 12/10.4 = 1.15\). 5. \(p(\mathbf{x} \mid \mathbf{y})\,p(\mathbf{y} \mid \mathbf{x})^s \propto p(\mathbf{x})\,p(\mathbf{y} \mid \mathbf{x})\,p(\mathbf{y} \mid \mathbf{x})^s = p(\mathbf{x})\,p(\mathbf{y} \mid \mathbf{x})^{s+1}\); taking gradients of the log gives the same guided score. 6. \(y = 1.5x + v\,\theta_c(x + uc) + b\), so \(\partial y/\partial u = v\,\theta_c\,c = 0\) because \(v = 0\), while \(\partial y/\partial v = \theta_c(x + uc) = 1.5x \ne 0\). If \(\theta_c = 0\) too, then \(\partial y/\partial v = \theta_c(x + uc) = 0\), \(\partial y/\partial u = v\theta_c c = 0\) and \(\partial y/\partial \theta_c = v(x + uc) = 0\): each gradient contains another parameter that is zero, so \(u\), \(v\) and \(\theta_c\) stay at 0 forever and only \(b\) learns. The branch could never learn to use \(c\). This is why the copy keeps the pretrained weights and only the connections start at zero.

Key takeaways

  • Text enters the U-Net through cross-attention (227.1): queries from image positions, keys and values from text tokens (SD 1.5: 4,096 queries, 77 keys of width 768 at the top level).
  • Bayes’ rule for scores (227.2) splits the conditional score into the unconditional score plus a classifier gradient. Scaling that gradient is guidance.
  • Classifier guidance (227.4) needs a classifier trained on noisy images and samples \(p(\mathbf{x})p(\mathbf{y} \mid \mathbf{x})^s\) (227.5).
  • Classifier-free guidance (227.7) uses the same network with and without the condition, trained with ~10% condition dropout (227.12); it costs two evaluations per step.
  • Higher guidance raises precision and IS and lowers recall and diversity; FID is best at small guidance, people prefer more. Very high scales saturate and amplify errors; rescale (227.13), thresholding and limited intervals help.
  • ControlNet (227.14) adds spatial control: locked model, trainable encoder copy, zero convolutions so training starts exactly at the pretrained model, and gradients still flow (227.15).

Key formulas

\[ \tilde{\epsilon} = \epsilon_u + w(\epsilon_c - \epsilon_u), \qquad \hat{\epsilon} = \epsilon_\theta - s\sqrt{1-\bar\alpha_t}\,\nabla_{\mathbf{x}_t}\log p_\phi(\mathbf{y} \mid \mathbf{x}_t) \]
\[ \tilde p_w(\mathbf{x}) \propto p(\mathbf{x})\,p(\mathbf{y} \mid \mathbf{x})^{w}, \qquad \mathbf{y}_c = \mathcal{F}(\mathbf{x}; \Theta) + \mathcal{Z}\big(\mathcal{F}(\mathbf{x} + \mathcal{Z}(\mathbf{c}); \Theta_c)\big) \]

References for this session

  1. Dhariwal & Nichol (2021). Diffusion Models Beat GANs on Image Synthesis
  2. Ho & Salimans (2022). Classifier-Free Diffusion Guidance
  3. Zhang et al. (2023). Adding Conditional Control to Text-to-Image Diffusion Models
  4. Dhariwal & Nichol (2021). Diffusion Models Beat GANs on Image Synthesis
  5. Ho & Salimans (2022). Classifier-Free Diffusion Guidance
  6. Zhang et al. (2023). Adding Conditional Control to Text-to-Image Diffusion Models
  7. Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models
  8. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  9. Ho et al. (2020). Denoising Diffusion Probabilistic Models
  10. Song et al. (2020). Denoising Diffusion Implicit Models
  11. Song et al. (2020). Score-Based Generative Modeling through Stochastic Differential Equations
  12. Lin, Liu, Li & Yang (2023). Common Diffusion Noise Schedules and Sample Steps are Flawed
  13. Saharia et al. (2022). Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
  14. Kynkäänniemi, Aittala, Karras, Laine, Aila & Lehtinen (2024). Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models
  15. Mou et al. (2023). T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
  16. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 10: Any-to-Any Models