Guidance and conditioning: classifier guidance, classifier-free guidance, ControlNet
A diffusion model that only learned p(x | prompt) follows the prompt weakly. This session shows how the prompt enters the network (cross-attention), how classifier guidance and classifier-free guidance push samples toward the prompt, why a large guidance scale gives sharp but less varied images, and how ControlNet adds spatial control (edges, pose, depth) to a frozen model with zero-initialized convolutions.
What you will learn
- How text conditioning enters a diffusion U-Net through cross-attention: queries from image tokens, keys and values from text tokens, with the real Stable Diffusion 1.5 shapes
- Bayes' rule for scores, and how classifier guidance adds the gradient of a noisy classifier to the noise prediction
- Classifier-free guidance: one network trained with label dropout, two forward passes per step, and the formula eps_u + w (eps_c - eps_u)
- The two conventions for the guidance scale (Ho and Salimans w versus the Stable Diffusion scale) and how to convert between them
- Why high guidance gives sharp but less varied images, shown exactly on a 1-D toy and with the published FID, IS, precision and recall numbers
- What goes wrong at very high guidance (saturation, error amplification) and the common fixes: guidance rescale, thresholding, limited intervals
- ControlNet: a locked model, a trainable copy, and zero convolutions that make training start exactly at the pretrained model
Why this matters
Type “a red car on a beach” into Stable Diffusion, and the model returns a red car on a beach. Two separate mechanisms make that happen. Conditioning lets the network read the prompt. Guidance makes the sampler obey it strongly. Without guidance, the same model gives blurry, off-topic images much of the time.
In session 225 and session 226 we learned to sample from \(p(\mathbf{x})\): any image that looks like training data. Real products need \(p(\mathbf{x} \mid \mathbf{y})\), where \(\mathbf{y}\) is a class label, a text prompt, an edge map, a pose skeleton or a depth map. This session answers three questions:
- How does the condition enter the network? For text: cross-attention from image tokens to text tokens.
- How do we make samples follow the condition strongly? Classifier guidance (Dhariwal and Nichol, 2021) and classifier-free guidance (CFG) (Ho and Salimans, 2022). CFG is in almost every text-to-image and text-to-video model. The “guidance scale” slider in every image app is the CFG scale.
- How do we add a spatial condition to a big pretrained model without breaking it? ControlNet (Zhang, Rao and Agrawala, 2023): a trainable copy of the encoder, connected through convolutions initialized to zero.
The trade-off is the heart of the topic. Guidance moves probability mass toward samples that are “very clearly” the condition. That raises sample quality and prompt alignment, but it lowers diversity. In the original classifier-guidance paper, a scale of 10 on a conditional ImageNet model raised precision from 0.69 to 0.88 and lowered recall from 0.63 to 0.32. [Certain] You will see exactly why, on a toy problem you can solve by hand.
Conditioning: how the prompt gets into the network
A conditional diffusion model is a noise predictor with one extra input: \(\epsilon_\theta(\mathbf{x}_t, t, \mathbf{y})\). It is trained with the same loss as before, only now each training image comes with its label or caption \(\mathbf{y}\). The question is how \(\mathbf{y}\) is fed in. There are three common ways:
- Add it to the time embedding. A class label becomes a learned vector, added to the vector that encodes \(t\). Every residual block already receives the time embedding, so every block now also “knows” the class. This is how class-conditional ImageNet models work. [Certain]
- Concatenate it to the input. For spatial conditions with the same size as the image (a low-resolution image for super-resolution, a mask for inpainting), stack them as extra input channels. The Stable Diffusion inpainting model has 5 extra input channels: 4 for the encoded masked image and 1 for the mask. [Certain]
- Cross-attention. For text, which is a sequence of variable meaning, the image features attend to the text tokens. This is the main mechanism in Stable Diffusion 1.x, 2.x and SDXL. [Certain]
Cross-attention
Recall attention from session 20: queries ask, keys answer “how relevant am I”, values carry the content. In self-attention, Q, K and V all come from the same sequence. In cross-attention, the queries come from one sequence and the keys and values from another. In the diffusion U-Net, the queries come from the image features, and the keys and values come from the text encoder output:
- Read the symbols
- \(\mathbf{H}\)
- the image feature map flattened into tokens: \(h \cdot w\) rows (one per spatial position), \(C\) columns (channels)
- \(\tau(\mathbf{y})\)
- the text encoder output for prompt \(\mathbf{y}\): one vector per text token
- \(W_Q, W_K, W_V\)
- learned projection matrices; \(W_K, W_V\) map the text width to the image width
- \(d\)
- the dimension of one attention head
- result
- for every image position, a weighted mix of text-token values, added back to \(\mathbf{H}\) through a residual connection
So each spatial position of the noisy image asks “which words are about me?” and pulls in their meaning. The position where the car should go attends to “red” and “car”; the bottom of the image attends to “beach”. This is also why attention maps from cross-attention layers can be used for editing (change one word, keep the layout).
From the model’s unet/config.json: cross_attention_dim = 768, block_out_channels = [320, 640, 1280, 1280], latent sample_size = 64. [Certain] The text encoder is CLIP ViT-L/14 with a maximum of 77 tokens, so \(\tau(\mathbf{y})\) is \(77 \times 768\). [Certain]
At the first level the latent is \(64 \times 64 = 4{,}096\) positions with 320 channels, so \(\mathbf{Q}\) is \(4{,}096 \times 320\), and \(W_K, W_V\) are \(768 \times 320\). The config value attention_head_dim = 8 is used as the number of heads in this old config, so each head has \(320/8 = 40\) dimensions. [Likely] One head’s score matrix \(\mathbf{Q}\mathbf{K}^\top\) is \(4{,}096 \times 77 = 315{,}392\) numbers. The lower levels have \(32 \times 32 = 1{,}024\) and \(16 \times 16 = 256\) query tokens. Compare self-attention at level 1: \(4{,}096 \times 4{,}096\), which is 53 times bigger. Cross-attention is cheap because the text is short.
Larger models change only the numbers: SDXL has cross_attention_dim = 2048 because it concatenates two text encoders. [Certain] Diffusion transformers (next session, session 228) often replace cross-attention with joint attention over image and text tokens, or with adaptive LayerNorm for class labels and time.
Why conditioning alone is not enough
Train \(\epsilon_\theta(\mathbf{x}_t, t, \mathbf{y})\) perfectly and sample from it. You get exact samples of \(p(\mathbf{x} \mid \mathbf{y})\). That sounds like what we want, but in practice the samples look poor: many images match the prompt only loosely, and the overall quality is low. The original numbers show it. A class-conditional ImageNet 256 × 256 model without guidance had FID 10.94; the same model with classifier guidance (scale 1) had FID 4.59. [Certain] (FID, the Fréchet Inception Distance, compares statistics of generated and real images; lower is better.)
Two reasons. First, the model is not perfect, and the condition is a weak signal compared to the strong task of denoising, so the network partly ignores it. Second, even the true \(p(\mathbf{x} \mid \text{“dog”})\) contains many unclear, cluttered photos where the dog is small. People prefer the “typical, clear” dogs. In GANs this was solved with the truncation trick, and in language models with a temperature below 1 (session 79). Guidance is the diffusion version of low temperature: make the sample more clearly “of the class”.
The tool is Bayes’ rule. Take the log of \(p(\mathbf{x}_t \mid \mathbf{y}) = p(\mathbf{x}_t)\, p(\mathbf{y} \mid \mathbf{x}_t) / p(\mathbf{y})\) and the gradient with respect to \(\mathbf{x}_t\). The term \(p(\mathbf{y})\) does not depend on \(\mathbf{x}_t\), so it disappears:
The conditional score is the unconditional score plus the gradient of a classifier. The second term points toward images that the classifier finds more clearly of class \(\mathbf{y}\). If we multiply that term by a number larger than 1, we “over-condition”. Every method in this session is a variation of this idea.
Classifier guidance
From session 226: a noise predictor is a scaled score. For the variance-preserving process \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon\),
Dhariwal and Nichol train a separate classifier \(p_\phi(\mathbf{y} \mid \mathbf{x}_t, t)\) on noisy images (the downsampling half of their U-Net, trained on the same noise distribution as the diffusion model). [Certain] Put (227.3) into (227.2), add a gradient scale \(s\), and turn the score back into a noise prediction:
- Read the symbols
- \(\epsilon_\theta\)
- the diffusion model’s noise prediction (it may be unconditional or already class-conditional)
- \(p_\phi(\mathbf{y} \mid \mathbf{x}_t, t)\)
- a classifier that sees the noisy image and the time step
- \(s\)
- the gradient scale; \(s = 0\) is no guidance, \(s = 1\) is plain Bayes’ rule
- \(\hat{\epsilon}\)
- the guided noise prediction; DDIM or DDPM then runs as usual with \(\hat{\epsilon}\) in place of \(\epsilon_\theta\)
What distribution does this sample? Since \(s \nabla \log p(\mathbf{y} \mid \mathbf{x}) = \nabla \log p(\mathbf{y} \mid \mathbf{x})^s\), the guided score belongs to
Raising a probability to a power \(s \gt 1\) makes it sharper: regions where the classifier is 99% sure keep almost all their weight, regions where it is 60% sure lose most of it. The paper says it directly: a larger scale “focuses more on the modes of the classifier”. [Certain] At \(s = 1\) on an unconditional model, the classifier gave the right class about 50% probability on final samples, and the images did not look like the class; at \(s = 10\) the probabilities went to nearly 100%. [Certain]
| ImageNet 256 × 256 (Dhariwal & Nichol, Table 4) | FID ↓ | IS ↑ | Precision ↑ | Recall ↑ |
|---|---|---|---|---|
| Unconditional, no guidance | 26.21 | 39.70 | 0.61 | 0.63 |
| Unconditional, scale 1 | 33.03 | 32.92 | 0.56 | 0.65 |
| Unconditional, scale 10 | 12.00 | 95.41 | 0.76 | 0.44 |
| Conditional, no guidance | 10.94 | 100.98 | 0.69 | 0.63 |
| Conditional, scale 1 | 4.59 | 186.70 | 0.82 | 0.52 |
| Conditional, scale 10 | 9.11 | 283.92 | 0.88 | 0.32 |
Read the table column by column. Precision (the share of generated images that fall inside the real-image distribution, a fidelity measure) goes up with the scale. Recall (the share of the real distribution that the generated images cover, a diversity measure) goes down. IS (Inception Score, high when each image is confidently one class and the classes are varied) rises a lot. FID, which punishes both bad images and missing diversity, has its best value at a moderate scale. [Certain]
The problems with classifier guidance
- You need a second model, trained on noisy images at every noise level. An off-the-shelf classifier (or CLIP) does not work well on very noisy inputs. [Likely]
- The classifier gradient can be adversarial: a small, unnatural change to \(\mathbf{x}_t\) can raise the classifier score without making the image more like the class. Image gradients of classifiers are famous for this.
- It needs a classifier for the condition. For free-form text prompts there is no simple classifier \(p(\text{prompt} \mid \mathbf{x})\).
- Each step needs a backward pass through the classifier, which costs extra time and memory.
Classifier-free guidance
Ho and Salimans asked: can we get the classifier gradient without a classifier? Use Bayes’ rule the other way round. The classifier implied by a conditional and an unconditional diffusion model is \(p(\mathbf{y} \mid \mathbf{x}_t) \propto p(\mathbf{x}_t \mid \mathbf{y}) / p(\mathbf{x}_t)\). Its gradient, written with noise predictions through (227.3), is
The difference between the conditional and the unconditional noise prediction is a classifier gradient (up to scale). Now apply guidance with this implicit classifier to the unconditional model, with scale \(w\), and simplify. Write \(\epsilon_c = \epsilon_\theta(\mathbf{x}_t, t, \mathbf{y})\) and \(\epsilon_u = \epsilon_\theta(\mathbf{x}_t, t, \varnothing)\):
- Read the symbols
- \(\varnothing\)
- the “null” condition: an empty prompt or a special null class id
- \(\epsilon_u, \epsilon_c\)
- unconditional and conditional noise predictions of the same network
- \(w\)
- the guidance scale: \(w = 0\) is unconditional, \(w = 1\) is plain conditional, \(w \gt 1\) extrapolates past the conditional prediction
- \(\epsilon_c - \epsilon_u\)
- the guidance direction: what the condition changes
We start from a point on the line through \(\epsilon_u\) and \(\epsilon_c\), and go \(w\) times the distance in the direction the condition points. It is extrapolation, not interpolation. The sampled distribution, if all scores were exact, would be
This is (227.5) with the implicit classifier. So CFG with scale \(w\) on a conditional model is classifier guidance with scale \(w - 1\) on the conditional model, or scale \(w\) on the unconditional one. [Certain] One caveat that matters for the next section: (227.11) is the target at each noise level separately. Running the sampler with the guided noise does not, in general, produce samples of \(\tilde p_w\) at \(t = 0\). [Certain for the toy below; the general statement is Likely]
Training: one network, two jobs
We need \(\epsilon_u\) and \(\epsilon_c\). Training two networks would double the cost. Instead, train one network and, with probability \(p_{\text{uncond}}\), replace the condition with \(\varnothing\):
(The “\(b\,\varnothing + (1-b)\,\mathbf{y}\)” only means: use \(\varnothing\) when \(b = 1\), else \(\mathbf{y}\).) Ho and Salimans tried \(p_{\text{uncond}} \in \{0.1, 0.2, 0.5\}\): 0.1 and 0.2 worked about equally well, and 0.5 was worse across the whole quality frontier. [Certain] Stable Diffusion 1.x dropped the text condition 10% of the time. [Certain] For text, \(\varnothing\) is usually the encoding of the empty string “”.
- input: data pairs \((\mathbf{x}_0, \mathbf{y})\), drop rate \(p_{\text{uncond}}\) (often 0.1), schedule \(\bar\alpha_t\)
- repeat
- sample \((\mathbf{x}_0, \mathbf{y})\), \(t \sim \text{Uniform}\{1, \dots, T\}\), \(\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\)
- with probability \(p_{\text{uncond}}\): \(\mathbf{y} \leftarrow \varnothing\) // empty prompt or null class
- \(\mathbf{x}_t \leftarrow \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon\)
- gradient step on \(\lVert \epsilon - \epsilon_\theta(\mathbf{x}_t, t, \mathbf{y}) \rVert^2\) // formula (227.12)
- until converged
- input: condition \(\mathbf{y}\), scale \(w\), time steps \(t_N \gt \dots \gt t_1\)
- \(\mathbf{x} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\)
- for \(i = N, \dots, 1\) do
- \([\epsilon_u, \epsilon_c] \leftarrow \epsilon_\theta([\mathbf{x}, \mathbf{x}], t_i, [\varnothing, \mathbf{y}])\) // one batch of 2: two forward passes of work
- \(\tilde{\epsilon} \leftarrow \epsilon_u + w(\epsilon_c - \epsilon_u)\) // formula (227.7)
- \(\hat{\mathbf{x}}_0 \leftarrow (\mathbf{x} - \sqrt{1 - \bar\alpha_{t_i}}\,\tilde{\epsilon}) / \sqrt{\bar\alpha_{t_i}}\)
- \(\mathbf{x} \leftarrow \sqrt{\bar\alpha_{t_{i-1}}}\,\hat{\mathbf{x}}_0 + \sqrt{1 - \bar\alpha_{t_{i-1}}}\,\tilde{\epsilon}\) // DDIM step, \(\bar\alpha_{t_0} = 1\)
- end for; return \(\mathbf{x}\)
Take a 2-number toy: \(\epsilon_u = (0.2, -0.1)\) and \(\epsilon_c = (0.5, 0.3)\). The guidance direction is \(\epsilon_c - \epsilon_u = (0.3, 0.4)\), with length 0.5. With \(w = 7.5\): \(\tilde{\epsilon} = (0.2 + 7.5 \cdot 0.3,\; -0.1 + 7.5 \cdot 0.4) = (2.45,\; 2.9)\). The guided prediction lies \(6.5 \times 0.5 = 3.25\) beyond the conditional one, far outside the range of values the network ever produced in training. Keep that in mind for the “saturation” problem below.
Cost: 50 DDIM steps with CFG need \(2 \times 50 = 100\) network evaluations, twice the unguided cost. With batching they run as one batch of 2, so latency grows less than 2× on a GPU that is not already full, but the FLOPs double. [Certain for FLOPs]
Watching Algorithm 2 run on 2-D data
To see the formula at work, we need a model whose \(\epsilon_u\) and \(\epsilon_c\) we know exactly. Take 2-D data with three classes A, B and C. Each class is a round Gaussian \(\mathcal{N}(\mu_k, s^2 \mathbf{I})\) with \(s = 0.55\), and the classes are equally likely. The noisy data \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\epsilon\) of one class is again Gaussian, with mean \(\sqrt{\bar\alpha_t}\,\mu_k\) and variance \(v_t = \bar\alpha_t s^2 + 1 - \bar\alpha_t\) per axis. So the best possible noise prediction (the one a perfect network would learn) has a closed form:
The conditional prediction for class \(\mathbf{y} = k\) is \(\epsilon_c = \epsilon_k\). The unconditional prediction does not know the class, so it averages the three class predictions, weighted by how likely each class is given the noisy point:
- Read the symbols
- \(\mu_k\)
- the center of class \(k\): A at \((-1.6, -0.9)\), B at \((1.6, -0.9)\), C at \((0, 1.8)\)
- \(v_t\)
- the variance of the noisy class data at step \(t\): almost 1 at the start (pure noise), \(s^2\) at the end
- \(r_k(\mathbf{x})\)
- the posterior probability of class \(k\) for the noisy point: the implicit classifier of (227.6)
Because \(\hat{\mathbf{x}}_0 = (\mathbf{x} - \sqrt{1-\bar\alpha_t}\,\epsilon)/\sqrt{\bar\alpha_t}\) is a straight-line (affine) function of \(\epsilon\), the CFG mix carries over to the clean-image guesses with the same weights:
This is what Figure 2 draws. For the tracked point, the blue dot is where the unconditional model thinks the clean point is, the green dot is where the conditional model thinks it is, and the orange dot is the guided guess, \(w\) times as far along the same line. With \(w \gt 1\) the orange dot lies beyond the green one, on the side away from the other classes.
- \(\mathbf{x} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) // 240 points
- for \(i = 20, \dots, 1\) do // \(t_i\) from 999 to 0
- \(\epsilon_u, \epsilon_c \leftarrow \epsilon(\mathbf{x}, t_i, \varnothing),\ \epsilon(\mathbf{x}, t_i, \mathbf{y})\) // (227.9), (227.8)
- \(\tilde\epsilon \leftarrow \epsilon_u + w(\epsilon_c - \epsilon_u)\) // (227.7)
- \(\hat{\mathbf{x}}_0 \leftarrow (\mathbf{x} - \sqrt{1-\bar\alpha_{t_i}}\,\tilde\epsilon)/\sqrt{\bar\alpha_{t_i}}\)
- \(\mathbf{x} \leftarrow \sqrt{\bar\alpha_{t_{i-1}}}\,\hat{\mathbf{x}}_0 + \sqrt{1-\bar\alpha_{t_{i-1}}}\,\tilde\epsilon\)
- return \(\mathbf{x}\)
How to read Figure 2
Press Play. Each DDIM step runs lines 3 to 6; the panel on the right shows the formula of the current line with the real numbers of the tracked point. Early on (\(\bar\alpha_t\) near 0) all three guesses \(\hat{\mathbf{x}}_0\) sit near the middle, because the noisy point says little. As the noise drops, they separate and the points fall into place. The tracked point starts at \((2.76, 1.13)\), near class B. Set \(w = 0\) (unconditional): it ends in class B, at \((2.60, 0.03)\). With \(w = 1\) it ends in class C at \((1.32, 2.34)\); with \(w = 3\) at \((1.12, 2.93)\); with \(w = 7.5\) at \((1.00, 3.60)\), above the class C circle. Now drag \(w\) while paused: the whole run is recomputed and you see the same step with the new scale. In the chart at the bottom, the spread keeps falling as \(w\) grows, while the distance to the class center is smallest near \(w = 1.5\) and then grows: the samples become narrow and pushed away from the other classes, “more C than real C data”.
Class C, \(w = 3\), the tracked point at step 11 of 20 (\(t = 473\), \(\bar\alpha_t = 0.1015\), so \(\sqrt{\bar\alpha_t} = 0.3186\) and \(\sqrt{1-\bar\alpha_t} = 0.9479\)). The point is \(\mathbf{x} = (2.409, 2.542)\). The exact predictions are \(\epsilon_u = (2.334, 2.250)\) and \(\epsilon_c = (2.458, 2.009)\), so the guidance direction is \(\epsilon_c - \epsilon_u = (0.124, -0.241)\).
Line 4: \(\tilde\epsilon = (2.334, 2.250) + 3 \cdot (0.124, -0.241) = (2.706, 1.526)\). Line 5: \(\hat{\mathbf{x}}_0 = \big((2.409, 2.542) - 0.9479 \cdot (2.706, 1.526)\big) / 0.3186 = (-0.488, 3.439)\). The guided guess is above class C’s center \((0, 1.8)\): guidance removes less noise along the second axis, so the point is pushed up, away from classes A and B below.
The final samples (all 240 points, class C) in numbers: with \(w = 0\), only 27.1% land in class C’s region (the region where C’s center is the nearest). With \(w = 1\), 99.6% do, with a spread (standard deviation per axis) of 0.48. With \(w = 3\): 100%, spread 0.33, mean at \(y = 2.47\). With \(w = 7.5\): 100%, spread 0.28, mean at \(y = 3.22\), far above the true center 1.8. The true class spread is 0.55; with 20 steps the sampler gives a bit less (0.48 at \(w = 1\); 0.54 with 200 steps). [Certain for this toy]
Negative prompts
Nothing forces \(\epsilon_u\) to use the empty prompt. If you replace it with the prediction for a negative prompt \(\mathbf{y}^-\) (“blurry, low quality, extra fingers”), formula (227.7) becomes \(\epsilon_{\mathbf{y}^-} + w(\epsilon_{\mathbf{y}} - \epsilon_{\mathbf{y}^-})\): the sampler moves toward the prompt and away from the negative prompt. In diffusers this is the negative_prompt argument; its embedding simply takes the place of the empty-prompt embedding in the batch. [Certain]
Why high guidance gives sharp but less varied images
Let us see the effect exactly on a problem small enough to compute. One-dimensional data, two classes with equal probability: class “+” is \(\mathcal{N}(+1, 0.5^2)\), class “−” is \(\mathcal{N}(-1, 0.5^2)\). Because everything is Gaussian, the true scores of the noisy distributions are known in closed form at every noise level. No network, no training error. We condition on class “+” and run the exact probability-flow ODE (the deterministic sampler of session 226) with the guided score \(s_u + w(s_c - s_u)\) on 400 evenly spaced starting points.
How to read Figure 3
At \(w = 0\) the samples cover both classes (unconditional). At \(w = 1\) they match the blue class-“+” curve: plain conditional sampling, and about 2% of samples still fall on the “−” side, exactly like real data of class “+”. Increase \(w\): the histogram moves away from the other class and gets narrower. Nothing ends up on the wrong side, but the spread (diversity) shrinks. Compare the orange histogram with the green dashed curve: the guided sampler pushes much farther than the “tilted” target (227.11) evaluated at \(t = 0\), because guidance acts at every noise level and the effects add up along the path. Press “Play the sampler” (with \(w = 3\)) to watch this happen. At \(\sigma = 4.7\) the guided cloud is already shifted: its mean is 2.74, against 0.94 for the plain conditional (\(w = 1\)) samples, because at high noise the guidance term moves the whole cloud. As \(\sigma\) falls, the class score pulls the samples back, but not all the way: the mean ends at 1.51 versus 0.99. The narrowing comes later: the guided spread is 0.96 times the plain one at \(\sigma = 4.7\), 0.89 times at \(\sigma = 2.8\), and 0.63 times from \(\sigma \approx 0.6\) on (0.31 versus 0.50 at the end).
The implicit classifier. For this data, \(p(+ \mid x) = 1/(1 + e^{-8x})\) at \(t = 0\) (the factor is \(2 \cdot 1 / 0.5^2 = 8\)). At \(x = 0.3\): \(p(+ \mid x) = 0.917\), the class score is \(s_c = -(0.3 - 1)/0.25 = 2.8\), and the unconditional score is \(s_u = 2.135\). The difference \(s_c - s_u = 0.665\) equals \(\frac{d}{dx}\log p(+ \mid x) = 8\,(1 - 0.917)\), as (227.6) says. With \(w = 3\), the guided score is \(2.135 + 3 \cdot 0.665 = 4.131\): the point is pushed toward larger \(x\) much harder than the plain conditional score 2.8.
The samples. Computed with the ODE (and checked in Python with the same algorithm):
| \(w\) | mean | std (diversity) | share classified “+” |
|---|---|---|---|
| 0 | 0.00 | 1.12 | 50% |
| 1 | 0.99 | 0.50 | 97.8% |
| 3 | 1.51 | 0.31 | 100% |
| 7.5 | 2.06 | 0.25 | 100% |
From \(w = 1\) to \(w = 7.5\), the spread halves and the mean moves to twice the true class center. The samples are “more class +” than the real class + data ever is. In images, that is the over-saturated, over-sharpened, “too perfect” look of very high CFG. The tilted target (227.11) at \(t = 0\) with \(w = 7.5\) would have mean 1.08 and std 0.44: the real sampler goes much further. [Certain for this toy]
The published trade-off curves
Ho and Salimans swept the guidance strength on ImageNet 64 × 64 and measured FID and IS. Figure 4 plots their Table 1. Their best FID came from small guidance (\(w' = 0.1\) to \(0.3\)), and the best IS from strong guidance (\(w' \ge 4\)). [Certain] Note how small the FID-optimal scale is, compared with the 7.5 used for text-to-image. FID rewards diversity; people judging single images prefer more guidance. [Likely]
How to read Figure 4
Moving right along a curve means more guidance. IS keeps rising: each image is more clearly one class. FID first drops (better images), reaches its minimum at \(w' = 0.1\) for \(p_{\text{uncond}} = 0.1\), then rises steeply as diversity collapses. Switch to \(p_{\text{uncond}} = 0.5\): the whole curve is worse, because half the training went to the unconditional task.
What goes wrong at very high guidance, and the fixes
- Saturation and over-exposure. As the worked example showed, \(\tilde{\epsilon}\) leaves the range the network was trained on, so the implied \(\hat{\mathbf{x}}_0\) has values far outside the valid pixel range. Imagen introduced dynamic thresholding: at each step, clip \(\hat{\mathbf{x}}_0\) to a percentile of its absolute values and rescale. [Certain]
- Guidance rescale. Lin et al. rescale the guided prediction so its standard deviation matches the conditional one, then mix:
\[ \mathbf{x}_{\text{res}} = \mathbf{x}_{\text{cfg}} \cdot \frac{\operatorname{std}(\mathbf{x}_{c})}{\operatorname{std}(\mathbf{x}_{\text{cfg}})}, \qquad \mathbf{x}_{\text{final}} = \phi\, \mathbf{x}_{\text{res}} + (1 - \phi)\, \mathbf{x}_{\text{cfg}} \tag{227.13} \]They report that \(w = 7.5\), \(\phi = 0.7\) works well. [Certain] In diffusers this is
guidance_rescale, applied to the noise prediction. - Error amplification. A learned \(\epsilon_c - \epsilon_u\) has errors, and \(w\) multiplies them. In the PyTorch toy below, a small trained network gives the right answer at \(w = 3\) but at \(w = 7.5\) its samples are wider (std 1.53) instead of narrower (exact: 0.25 with the same sampler). [Certain, our run]
- Guidance only where it helps. Kynkäänniemi et al. apply guidance only in a middle range of noise levels: at the highest noise it mostly destroys diversity, at the lowest it does little. [Likely]
- Removing the second pass. Distilled models learn to output the guided prediction directly, with \(w\) as an input, so one forward pass per step is enough. Some open “guidance-distilled” models work this way. [Likely]
ControlNet: spatial control for a frozen model
Text says what to draw but is weak at saying where. “A dancer with her left arm raised” rarely gets the exact pose. ControlNet adds a spatial condition: an image of the same size that fixes the layout. Examples: Canny edges, a depth map, a normal map, straight lines (M-LSD), soft edges (HED), a segmentation map, a human pose skeleton (OpenPose), a scribble. [Certain]
The challenge: Stable Diffusion was trained on billions of images; a pose dataset may have only thousands. Fine-tuning the whole model on it would cause overfitting and forgetting. ControlNet’s answer has three parts.
- Lock the original network. Its weights \(\Theta\) never change.
- Make a trainable copy \(\Theta_c\) of the encoder blocks, initialized with the same weights. The copy is a strong, pretrained feature extractor from the first step.
- Connect them with zero convolutions \(\mathcal{Z}\): \(1 \times 1\) convolutions whose weight and bias start at exactly zero.
- Read the symbols
- \(\mathcal{F}(\cdot; \Theta)\)
- a pretrained block with locked weights; \(\mathbf{x}\) is its input feature map
- \(\mathcal{F}(\cdot; \Theta_c)\)
- the trainable copy of the same block
- \(\mathbf{c}\)
- the condition features (edge map, pose, depth), encoded to the latent size
- \(\mathcal{Z}(\cdot; \Theta_{z})\)
- a \(1 \times 1\) convolution with weight and bias initialized to 0
- \(\mathbf{y}_c\)
- the output: the original output plus a learned correction
At the first training step both \(\mathcal{Z}\) terms are zero, so \(\mathbf{y}_c = \mathcal{F}(\mathbf{x}; \Theta)\) exactly. The model starts as the unchanged pretrained model and can only improve from there. No random noise is injected into the features at the start of training. [Certain]
Why zero does not mean “stuck”
If a layer’s weights are zero, can it still learn? Yes, if its input is not zero. For a \(1 \times 1\) convolution \(\mathbf{z} = W\mathbf{h} + \mathbf{b}\) at one position,
With \(W = 0\), the first two are non-zero (the copy’s features \(\mathbf{h}\) are not zero), but the third is zero. So at step 1 only the output zero convolution moves. From step 2 on, \(W \ne 0\), gradients flow into the trainable copy and into the input zero convolution. Figure 5 shows this on a toy you can follow by hand.
\(y = 1.5x + v\,\theta_c\,(x + u\,c) + b\); bold boxes have a non-zero gradient at this step
How to read Figure 5
Step 0: the output is exactly the locked model’s output, and the loss equals the base model’s loss, 0.375. Only \(v\) and \(b\) (the output zero convolution) have non-zero gradients; \(u\) and \(\theta_c\) have gradient exactly 0, as (227.15) predicts. From step 1 on, all parameters move. The last step compares with a random (non-zero) initialization: the loss at step 0 is 2.09, much worse than the pretrained model, which is the damage zero convolutions avoid.
ControlNet on Stable Diffusion
The details from the paper [Certain]:
- The trainable copy covers the 12 encoder blocks and the middle block of the SD U-Net (4 resolutions, 64, 32, 16 and 8, three blocks each). Their outputs, through zero convolutions, are added to the 12 skip connections and the middle block of the locked U-Net.
- The condition image (512 × 512) is turned into a 64 × 64 feature map by a tiny encoder: four convolutions with 4 × 4 kernels and stride 2, with 16, 32, 64 and 128 channels.
- The locked part needs no gradients, so training costs only about 23% more GPU memory and 34% more time per iteration than fine-tuning SD alone (one A100 40 GB).
- 50% of text prompts are replaced by empty strings during training, so the ControlNet learns to read the meaning of the condition image itself.
- Sudden convergence: the model does not learn the condition gradually; it suddenly starts to follow it, usually in fewer than 10K steps.
- Training works with small (under 50k) and large (over 1M) datasets. Several ControlNets (for example pose and depth) can be combined by adding their outputs.
- input: latent \(\mathbf{x}_t\), step \(t\), text embeddings \(\tau(\mathbf{y}), \tau(\varnothing)\), condition image \(\mathbf{c}_{\text{img}}\), CFG scale \(w\), control strength \(\lambda\)
- \(\mathbf{c} \leftarrow E(\mathbf{c}_{\text{img}})\) // tiny conv encoder, \(512 \to 64\), computed once
- for each text input \(e \in \{\tau(\varnothing), \tau(\mathbf{y})\}\) do // one batch of 2
- \(\{\mathbf{r}_1, \dots, \mathbf{r}_{12}, \mathbf{r}_{\text{mid}}\} \leftarrow \text{ControlNet}(\mathbf{x}_t, t, e, \mathbf{c})\) // trainable copy + zero convs, (227.14)
- \(\epsilon_e \leftarrow \text{U-Net}(\mathbf{x}_t, t, e;\ \text{skip}_k \mathrel{+}= \lambda\,\mathbf{r}_k,\ \text{mid} \mathrel{+}= \lambda\,\mathbf{r}_{\text{mid}})\)
- end for
- \(\tilde{\epsilon} \leftarrow \epsilon_u + w(\epsilon_c - \epsilon_u)\); take the sampler step as in Algorithm 2
A subtle point: should the control be added to both the conditional and the unconditional branch of CFG? The paper reports that adding it to both removes the guidance when there is no prompt, while adding it only to \(\epsilon_c\) makes guidance too strong; their fix (“CFG resolution weighting”) adds it to \(\epsilon_c\) with a weight \(64/h_i\) that depends on the block resolution \(h_i\). [Certain] Common libraries add the control to both branches by default. [Likely]
Lighter alternatives exist. T2I-Adapter trains a small separate network whose features are added to the U-Net encoder, with far fewer parameters than a full encoder copy. [Likely] The same zero-initialization trick appears in many places: the SD inpainting model initialized its 5 new input channels with zero weights [Certain], and LoRA starts one of its two matrices at zero (session 57).
Code: classifier-free guidance and a zero-convolution block in PyTorch
The first program trains a tiny class-conditional diffusion model on the 1-D, two-class data of Figure 3, with label dropout (Algorithm 1), then samples with CFG and DDIM (Algorithm 2). It runs in about 20 seconds on a laptop CPU and is deterministic.
import math, torch, torch.nn as nn torch.manual_seed(0) # 1) toy data: two classes on a line, each N(+-1, 0.5^2) def data(n): c = torch.randint(0, 2, (n,)) x = (2 * c - 1).float() * 1.0 + 0.5 * torch.randn(n) return x[:, None], c T = 1000 betas = torch.linspace(1e-4, 0.02, T) abar = torch.cumprod(1 - betas, 0) # alpha-bar_t class Eps(nn.Module): # eps_theta(x_t, t, c); label 2 = "no label" def __init__(s, h=256): super().__init__() s.emb = nn.Embedding(3, 16) s.net = nn.Sequential(nn.Linear(1 + 16 + 16, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 1)) def forward(s, x, t, c): f = torch.exp(-math.log(1e4) * torch.arange(8) / 8) * t[:, None].float() return s.net(torch.cat([x, f.sin(), f.cos(), s.emb(c)], 1)) model = Eps() opt = torch.optim.AdamW(model.parameters(), lr=1e-3) sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, 6000) P_UNCOND = 0.1 for step in range(6000): # 2) training with label dropout, (227.12) x0, c = data(512) c = torch.where(torch.rand(512) < P_UNCOND, torch.full_like(c, 2), c) t = torch.randint(0, T, (512,)) eps = torch.randn_like(x0) xt = abar[t, None].sqrt() * x0 + (1 - abar[t, None]).sqrt() * eps loss = ((model(xt, t, c) - eps) ** 2).mean() opt.zero_grad(); loss.backward(); opt.step(); sched.step() @torch.no_grad() def sample(w, label=1, n=2000, steps=50): # 3) DDIM sampling with CFG g = torch.Generator().manual_seed(1) x = torch.randn(n, 1, generator=g) ts = torch.linspace(T - 1, 0, steps).long() for i, t in enumerate(ts): tt = t.repeat(n) x2, t2 = torch.cat([x, x]), torch.cat([tt, tt]) # one batch, two halves c2 = torch.cat([torch.full((n,), 2), torch.full((n,), label)]) e_u, e_c = model(x2, t2, c2).chunk(2) e = e_u + w * (e_c - e_u) # formula (227.7) x0 = (x - (1 - abar[t]).sqrt() * e) / abar[t].sqrt() ab_prev = abar[ts[i + 1]] if i + 1 < steps else torch.tensor(1.0) x = ab_prev.sqrt() * x0 + (1 - ab_prev).sqrt() * e return x for w in [0.0, 1.0, 3.0, 7.5]: x = sample(w) print(f"w={w:4}: mean {x.mean():.2f} std {x.std():.2f} share x>0 {(x > 0).float().mean():.3f}")
Our run (PyTorch 2.15, CPU) printed:
w= 0.0: mean 0.03 std 1.11 share x>0 0.517 w= 1.0: mean 0.99 std 0.49 share x>0 0.980 w= 3.0: mean 1.47 std 0.36 share x>0 1.000 w= 7.5: mean 2.70 std 1.53 share x>0 1.000
Compare with the exact values from the same DDIM sampler with the true noise predictions: 0.02 / 1.09, 1.00 / 0.47, 1.51 / 0.29, 2.06 / 0.23 (mean / std). Up to \(w = 3\) the small network is close. At \(w = 7.5\), its small errors in \(\epsilon_c - \epsilon_u\) are multiplied by 7.5, and the samples spread out instead of tightening. Exact numbers can change slightly with the PyTorch version or hardware.
The second program builds a ControlNet-style block, formula (227.14), and checks the two facts from Figure 5: the output equals the locked block at initialization, and only the output zero convolution gets a gradient at step 0.
import copy, torch, torch.nn as nn torch.manual_seed(0) def zero_conv(ch): z = nn.Conv2d(ch, ch, 1); nn.init.zeros_(z.weight); nn.init.zeros_(z.bias); return z class ControlledBlock(nn.Module): """y_c = F(x) + Z2(F_copy(x + Z1(c))) -- formula (227.14)""" def __init__(s, block, ch): super().__init__() s.locked = block.requires_grad_(False) # frozen original weights s.copy = copy.deepcopy(block).requires_grad_(True) # trainable copy, same start s.z1, s.z2 = zero_conv(ch), zero_conv(ch) def forward(s, x, c): return s.locked(x) + s.z2(s.copy(x + s.z1(c))) block = nn.Sequential(nn.Conv2d(8, 8, 3, padding=1), nn.SiLU(), nn.Conv2d(8, 8, 3, padding=1)) cb = ControlledBlock(block, 8) x, c = torch.randn(2, 8, 16, 16), torch.randn(2, 8, 16, 16) y = cb(x, c) print((y - block(x)).abs().max().item()) # 0.0: the pretrained model is unchanged y.pow(2).mean().backward() print(cb.z2.weight.grad.abs().sum().item() > 0, # True: the output zero conv learns first cb.z1.weight.grad.abs().sum().item(), # 0.0 at step 0 cb.copy[0].weight.grad.abs().sum().item()) # 0.0 at step 0
It prints 0.0, then True 0.0 0.0.
Where this is used
| System or task | Conditioning and guidance | Notes |
|---|---|---|
| Stable Diffusion 1.x / 2.x, SDXL | Cross-attention to frozen CLIP text encoders; CFG with the empty prompt as \(\varnothing\); diffusers default scale 7.5 with 50 steps | 10% caption dropout in SD 1.x training [Certain] |
| Class-conditional ImageNet models (ADM, DiT) | Class embedding added to the time embedding; classifier guidance (ADM-G) or CFG | The standard benchmark for the FID / IS trade-off |
| Controlled image generation and editing | ControlNet with edges, depth, pose, segmentation; several ControlNets added together | Layout from the condition, content from the prompt |
| Text-to-video generation | Cross- or joint attention to text, CFG on every frame’s latent; often a lower scale than for images [Likely] | Too much guidance shows as flicker and over-saturated colors over time [Speculation] |
| Pose- or depth-controlled video (motion transfer) | ControlNet-style branches fed with per-frame pose skeletons or depth maps [Likely] | Pose extracted from a source video drives a new character |
| Synthetic training data for video tasks (action recognition, segmentation, key-moment detection) | Pose or segmentation-map conditioning to create labeled frames; guidance scale tuned for diversity, not beauty | High CFG reduces diversity, which hurts a training set [Likely] |
| Autoregressive image generators and LLMs | CFG on logits: \(\ell_u + w(\ell_c - \ell_u)\) between conditional and unconditional next-token logits [Likely] | The same idea outside diffusion (session 79) |
Common mistakes
- Mixing the two scale conventions. A paper’s \(w' = 3\) is a diffusers
guidance_scale = 4. Always check whether 1 means “plain conditional” or 0 does. - Forgetting label dropout in training. Without it, \(\epsilon_\theta(\cdot, \varnothing)\) was never trained, and CFG combines a good prediction with garbage. You cannot add CFG to a model after training without fine-tuning.
- Using a different null embedding at sampling time than in training (for example zeros instead of the empty-prompt encoding). The unconditional branch must see exactly what it was trained with.
- Swapping \(\epsilon_u\) and \(\epsilon_c\) after
chunk(2). The order in the batch must match the order you unpack; a swap guides away from the prompt. - Turning the scale up to fix a bad prompt. Past a point, higher \(w\) only gives saturated, same-looking images and amplifies model errors. Use rescale, thresholding or a better prompt.
- Expecting \(w\) to transfer between samplers, step counts and models. The best scale depends on all three; re-tune it.
- Initializing the ControlNet connection randomly or fine-tuning the locked model too. Both damage the pretrained model at the start of training; keep the zero convolutions and the lock.
- A condition image that does not match the training preprocessing (a Canny map with different thresholds, a pose in a different keypoint format). ControlNets are trained on one specific detector’s output.
Interview questions
How does a text prompt enter a Stable Diffusion U-Net?basic
Write the classifier-free guidance formula and explain each term.basic
Derive classifier guidance from Bayes’ rule.core
Why is it called “classifier-free”? Where is the classifier hidden?core
How do you train a model so that CFG works, and what drop rate do you use?core
Why does high guidance give sharp but less diverse images?core
What is the cost of CFG at inference, and how can you reduce it?core
Images at guidance scale 15 look over-saturated. Why, and what can you do?advanced
Explain ControlNet and why its convolutions start at zero.core
Why make a copy of the encoder instead of training a small new network for the condition?advanced
Does sampling with CFG at scale \(w\) produce samples from \(p(\mathbf{x})^{1-w} p(\mathbf{x} \mid \mathbf{y})^w\)?advanced
Exercises
- A paper reports its best FID at \(w' = 0.3\) (Ho and Salimans convention). What
guidance_scaleis that in diffusers? And what is the default 7.5 in the paper’s convention? - Compute \(\tilde{\epsilon}\) for \(\epsilon_u = (1, 0)\), \(\epsilon_c = (0, 1)\) and \(w = 2\). What happens at \(w = 0.5\)?
- How many U-Net evaluations does one 30-step image need with CFG? With CFG and one ControlNet (count a ControlNet pass as one evaluation)?
- Assume \(p(x \mid c) = \mathcal{N}(1, 0.5^2)\) and \(p(x) = \mathcal{N}(0, 1.25)\) (a Gaussian stand-in for the mixture). Find the mean and standard deviation of \(p(x)^{1-w}p(x \mid c)^w\) for \(w = 3\). Hint: add precisions.
- Show that classifier guidance with scale \(s\) on a conditional model equals classifier guidance with scale \(s + 1\) on the unconditional model (exact scores).
- In Figure 5, explain why \(\partial L/\partial u = 0\) at step 0 but \(\partial L/\partial v \ne 0\). What would happen if \(\theta_c\) were also initialized to 0?
Answerscheck
Key takeaways
- Text enters the U-Net through cross-attention (227.1): queries from image positions, keys and values from text tokens (SD 1.5: 4,096 queries, 77 keys of width 768 at the top level).
- Bayes’ rule for scores (227.2) splits the conditional score into the unconditional score plus a classifier gradient. Scaling that gradient is guidance.
- Classifier guidance (227.4) needs a classifier trained on noisy images and samples \(p(\mathbf{x})p(\mathbf{y} \mid \mathbf{x})^s\) (227.5).
- Classifier-free guidance (227.7) uses the same network with and without the condition, trained with ~10% condition dropout (227.12); it costs two evaluations per step.
- Higher guidance raises precision and IS and lowers recall and diversity; FID is best at small guidance, people prefer more. Very high scales saturate and amplify errors; rescale (227.13), thresholding and limited intervals help.
- ControlNet (227.14) adds spatial control: locked model, trainable encoder copy, zero convolutions so training starts exactly at the pretrained model, and gradients still flow (227.15).
Key formulas
References for this session
- Dhariwal & Nichol (2021). Diffusion Models Beat GANs on Image Synthesis
- Ho & Salimans (2022). Classifier-Free Diffusion Guidance
- Zhang et al. (2023). Adding Conditional Control to Text-to-Image Diffusion Models
- Dhariwal & Nichol (2021). Diffusion Models Beat GANs on Image Synthesis
- Ho & Salimans (2022). Classifier-Free Diffusion Guidance
- Zhang et al. (2023). Adding Conditional Control to Text-to-Image Diffusion Models
- Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Ho et al. (2020). Denoising Diffusion Probabilistic Models
- Song et al. (2020). Denoising Diffusion Implicit Models
- Song et al. (2020). Score-Based Generative Modeling through Stochastic Differential Equations
- Lin, Liu, Li & Yang (2023). Common Diffusion Noise Schedules and Sample Steps are Flawed
- Saharia et al. (2022). Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
- Kynkäänniemi, Aittala, Karras, Laine, Aila & Lehtinen (2024). Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models
- Mou et al. (2023). T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 10: Any-to-Any Models