Latent diffusion and diffusion transformers (DiT)
Modern image and video generators do not run diffusion on pixels. They first compress the image with an autoencoder and run diffusion on the small latent, then decode once at the end. This session builds latent diffusion step by step, compares the U-Net backbone with the Diffusion Transformer (DiT), explains patchify and adaLN-Zero conditioning, and reads the DiT scaling results with real numbers.
What you will learn
- Why diffusing in a compressed latent space is cheaper and better: perceptual vs semantic compression, and how to count the 48× saving of the Stable Diffusion autoencoder
- How the autoencoder is trained (pixel, perceptual, adversarial and tiny KL losses), why the downsampling factor f = 4 to 8 works best, and why the latent scale 0.18215 matters
- The latent diffusion training and sampling algorithms, and where the encoder, denoiser and decoder each run
- The U-Net backbone with cross-attention to text, and its inductive biases
- The Diffusion Transformer: patchify, the token count T = (I/p)², adaLN and adaLN-Zero conditioning, and the 2C-channel output
- How to estimate the Gflops and parameters of a DiT from its config, and the DiT scaling result: compute, not parameter count, predicts FID
- How DiT led to text-to-image and video transformers (PixArt-α, MM-DiT, spacetime patches)
Why this matters
Almost every well-known image and video generator since 2022 does its diffusion in a small latent space, not on pixels. Stable Diffusion, SDXL, PixArt, Stable Diffusion 3, FLUX and Sora all follow this plan. And since 2023, most new ones also replaced the convolutional U-Net with a plain transformer, the Diffusion Transformer (DiT).
In session 225 and session 226 we learned to train a network that predicts noise, and to sample by removing noise step by step. Two problems remain. First, cost: a 512×512 RGB image has 786,432 numbers, and the network runs on all of them at every one of tens or hundreds of sampling steps. Second, most of those numbers are tiny details (exact texture, sensor noise) that the eye barely notices but that a pixel-space model must still spend capacity on. This session fixes both. We first compress the image with an autoencoder, then run diffusion on the small code. Then we change the denoiser from a U-Net to a transformer, which brings the scaling rules we know from LLMs to image generation.
The two papers behind this session are Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models (LDM, 2021), and Peebles & Xie, Scalable Diffusion Models with Transformers (DiT, 2022). We will read their key tables with real numbers.
Step 1: compress first, then diffuse
Think about what a generator must learn. Rombach et al. split it into two kinds of work. Perceptual compression removes high-frequency details that are hard to see but expensive to model. Semantic compression is the real creative part: which objects are in the scene, where, and how they relate. A pixel-space diffusion model does both jobs with one network. LDM gives the first job to an autoencoder that is trained once, and the second job to the diffusion model. [Certain]
The autoencoder has an encoder \(\mathcal{E}\) and a decoder \(\mathcal{D}\). The encoder shrinks the height and width by a downsampling factor \(f\) and outputs \(c\) channels:
- Read the symbols
- \(\mathbf{x}\)
- the image, \(H \times W\) pixels with 3 colors, scaled to \([-1, 1]\)
- \(\mathcal{E}, \mathcal{D}\)
- the encoder and decoder of the autoencoder; both are convolutional networks, frozen during diffusion training
- \(f\)
- the downsampling factor: each latent position covers an \(f \times f\) block of pixels
- \(c\)
- the number of latent channels (4 in Stable Diffusion 1.x and 2.x and SDXL)
- \(s\)
- a fixed scale factor that makes the latent values have standard deviation near 1 (explained below)
- \(\mathbf{z}_0\)
- the clean latent: the “data” for the diffusion model
The number of values shrinks by
Stable Diffusion v1 uses \(f = 8\) and \(c = 4\). [Certain] A 512×512 image has \(512 \cdot 512 \cdot 3 = 786{,}432\) values. Its latent is \(64 \times 64 \times 4 = 16{,}384\) values. The ratio is \(3 \cdot 8^2 / 4 = 48\). For the 256×256 images used by DiT, the latent is \(32 \times 32 \times 4 = 4{,}096\) values instead of \(196{,}608\). Again 48 times fewer.
Note that the spatial grid shrinks by \(f^2 = 64\), but the channel count grows from 3 to 4. So the latent is not 64 times smaller, “only” 48. Newer autoencoders (SD3, FLUX) keep \(f = 8\) but use \(c = 16\) to keep more detail, which gives \(3 \cdot 64/16 = 12\). [Likely]
How the autoencoder is trained
The autoencoder is a VAE-like network (session 223) but tuned for sharp reconstructions, much like VQGAN (session 224). Its loss mixes a pixel loss, a perceptual loss (distance between deep features of \(\mathbf{x}\) and \(\hat{\mathbf{x}}\)), a patch-based adversarial loss (a small discriminator that judges local realism), and a tiny regularizer on the latent:
The KL weight is very small (about \(10^{-6}\) in the released configs [Likely]). The KL term only stops the latent from drifting to huge values; it does not force the latent to be a nice Gaussian like in a real VAE. LDM also has a second option, a VQ layer instead of the KL term (“VQ-reg”). [Certain]
The scale factor \(s\)
Because the KL term is so weak, the raw latent has some arbitrary spread. The noise schedule from session 225 assumes data with a standard deviation near 1: at step \(t\), the signal-to-noise ratio is \(\bar\alpha_t / (1 - \bar\alpha_t)\) times the data variance. If the latent is 5 times too large, every noise level is effectively 25 times weaker than planned. So LDM measures the standard deviation of the latents on a batch of images and multiplies by \(s = 1/\text{std}\). For the Stable Diffusion 1.x VAE, \(s = 0.18215\); for the SDXL VAE, \(s = 0.13025\). [Certain] Forgetting \(s\), or dividing by it twice at decode time, is a classic bug.
How much to compress: the factor \(f\)
LDM tried \(f \in \{1, 2, 4, 8, 16, 32\}\) on ImageNet. With \(f = 1\) or 2 (almost pixels), training is slow. With \(f = 32\), quality stops improving early: the autoencoder has thrown away detail that no diffusion model can bring back. \(f = 4\) to 16 is the good range, with \(f = 4\) and 8 best overall. After 2 million steps, LDM-8 had an FID about 38 points better than the pixel-space LDM-1. [Certain] FID (Fréchet Inception Distance) compares statistics of deep features of real and generated images; lower is better.
Figure 1 lets you play with \(H\), \(f\), \(c\) and the patch size \(p\) that the transformer will use later.
How to read Figure 1
Start at the DiT setting (256, \(f = 8\), \(c = 4\), \(p = 2\)): 48× compression and 256 tokens. Now set \(f = 1\): the same transformer on raw pixels would need 16,384 tokens, and each attention matrix would be 4,096 times larger. Set \(c = 16\) to see how modern VAEs trade compression for detail. Try 1024 with \(f = 8\), \(p = 2\): 4,096 tokens, which is why high-resolution DiTs are expensive.
Step 2: diffusion in the latent space
Once the autoencoder is frozen, latent diffusion is exactly the DDPM of session 225, with \(\mathbf{x}_0\) replaced by \(\mathbf{z}_0\). We add noise, \(\mathbf{z}_t = \sqrt{\bar\alpha_t}\,\mathbf{z}_0 + \sqrt{1 - \bar\alpha_t}\,\varepsilon\), and train the denoiser to predict \(\varepsilon\):
- Read the symbols
- \(\varepsilon_\theta\)
- the denoiser (a U-Net or a DiT); it sees the noisy latent, the step and the condition
- \(y\)
- the condition: a class label, a text prompt, a depth map, ...
- \(\tau_\phi\)
- the condition encoder, for example a frozen CLIP text encoder that turns the prompt into a sequence of vectors
- \(t\)
- the noise step, drawn uniformly from \(1, \dots, 1000\)
Nothing about the diffusion math changes. That is the beauty of the design: the autoencoder is a plug-in. You can train it once and reuse it for many diffusion models. DiT, for example, did not train its own autoencoder at all; it used the off-the-shelf Stable Diffusion VAE. [Certain] Sampling also stays the same (DDPM, DDIM, or the fast solvers of session 226), and only at the very end do we call the decoder once.
- input: a batch of images \(\mathbf{x}\) and conditions \(y\); frozen \(\mathcal{E}\), scale \(s\); denoiser \(\varepsilon_\theta\); schedule \(\bar\alpha_1, \dots, \bar\alpha_{1000}\)
- \(\mathbf{z}_0 \leftarrow s \cdot \mathcal{E}(\mathbf{x})\) // no gradient; often pre-computed once for the whole dataset
- with probability 0.1, replace \(y\) by the empty condition \(\varnothing\) // enables classifier-free guidance
- \(t \sim \text{Uniform}\{1, \dots, 1000\}\), \(\;\varepsilon \sim \mathcal{N}(\mathbf{0}, I)\)
- \(\mathbf{z}_t \leftarrow \sqrt{\bar\alpha_t}\,\mathbf{z}_0 + \sqrt{1 - \bar\alpha_t}\,\varepsilon\)
- \(\mathcal{L} \leftarrow \lVert \varepsilon - \varepsilon_\theta(\mathbf{z}_t, t, \tau_\phi(y)) \rVert^2\) // formula (228.4)
- update \(\theta\) with AdamW; update an EMA copy of \(\theta\) // sample with the EMA weights
Watching the training step on a toy latent space
To see Algorithm 1 run with every number visible, we need a toy world. Our “images” are 16×16 RGB pictures in two classes. Sun: a yellow disc on a blue sky, on the left or on the right. Sea: light sky over dark water, with the horizon high or low. The toy encoder uses \(f = 4\) and \(c = 4\): for each 4×4 block of pixels it keeps the mean red, green and blue, plus one “edge” number that says how much the block gets brighter or darker from top to bottom, all multiplied by 5 (a real encoder also has some arbitrary output scale). So 768 pixel values become 64 latent values: \(3 \cdot 4^2 / 4 = 12\) times fewer, formula (228.2). The decoder paints each block back from its four numbers.
The round edge of the sun does not survive this encoder: the reconstruction error (mean squared, pixels in \([-1, 1]\)) is 0.0845 for the sun images and only 0.0069 for the sea images, whose horizon is a straight line that the edge channel can describe. That is the “\(f\) too large” problem in miniature. The raw latents have standard deviation 2.305, so \(s = 1/2.305 = 0.434\).
We also need a denoiser. Instead of training a network, we use the perfect one. When the data is a small set of known latents \(\mu_1, \dots, \mu_4\), each with a little Gaussian spread \(\sigma_d\) (we use 0.1), the noise prediction with the lowest possible loss (228.4) can be written down exactly. It first guesses which training latent \(\mathbf{z}_t\) came from, then removes the noise:
- Read the symbols
- \(\mu_k\)
- the scaled latent of training image \(k\); the sum runs over the images of class \(y\), or over all four when \(y = \varnothing\)
- \(w_k\)
- the probability that \(\mathbf{z}_t\) came from image \(k\): how well \(\mathbf{z}_t\) fits \(\sqrt{\bar\alpha_t}\,\mu_k\) plus noise of variance \(v_t\)
- \(\hat{\mathbf{z}}_0\)
- the best guess of the clean latent, \(\E[\mathbf{z}_0 \mid \mathbf{z}_t, y]\)
- \(\varepsilon^\ast\)
- the best guess of the noise; a trained \(\varepsilon_\theta\) tries to get as close as possible to this function
A real \(\varepsilon_\theta\) never sees this formula: it must learn the same mapping from millions of examples, for data far too large to list. But using the perfect one lets us watch the algorithm itself, with the learning problem already solved. Figure 2 runs Algorithm 1 on 8 random training examples. Every number is computed in your browser with the 1000-step linear schedule of DiT.
- \(\mathbf{z}_0 \leftarrow s \cdot \mathcal{E}(\mathbf{x})\) // 768 values \(\to\) 64
- with probability 0.1: \(y \leftarrow \varnothing\)
- \(t \sim \text{Uniform}\{0, \dots, 999\}\), \(\varepsilon \sim \mathcal{N}(\mathbf{0}, I)\)
- \(\mathbf{z}_t \leftarrow \sqrt{\bar\alpha_t}\,\mathbf{z}_0 + \sqrt{1 - \bar\alpha_t}\,\varepsilon\)
- \(\hat\varepsilon \leftarrow \varepsilon_\theta(\mathbf{z}_t, t, y)\), \(\mathcal{L} \leftarrow \operatorname{mean}(\varepsilon - \hat\varepsilon)^2\) // (228.4), with (228.5) as \(\varepsilon_\theta\)
How to read Figure 2
With seed 1 the first example is “sea, high horizon”. Its coin is \(u = 0.003 \lt 0.1\), so the label is dropped. It gets \(t = 527\), where \(\bar\alpha_t = 0.0588\): the decoded \(\mathbf{z}_t\) is mostly noise, and the signal-to-noise ratio is \(0.0588/0.9412 = 0.062\). The loss is 0.0021. Most dots sit near 0: the denoiser is perfect. Now pick seed 3 and step to example 5: \(t = 33\) and the loss is 0.1375, the largest one. At small \(t\) the noise is so small that it hides inside the natural spread \(\sigma_d\) of the data, and nobody can tell them apart. Finally choose “forget s”: the latent has standard deviation 2.305 instead of 1, and at the same \(t = 527\) the ratio becomes 0.332, \(2.305^2 = 5.3\) times more signal than the schedule planned. This is the bug from the scale section, now with numbers.
- input: condition \(y\), guidance scale \(w\), number of steps \(K\), decoder \(\mathcal{D}\), scale \(s\)
- \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\) with the latent shape, e.g. \(4 \times 64 \times 64\) for a 512×512 image
- for \(k = K, \dots, 1\) (steps \(t_K \gt \dots \gt t_1\)):
- \(\hat\varepsilon \leftarrow \varepsilon_\theta(\mathbf{z}, t_k, \varnothing) + w\,\big(\varepsilon_\theta(\mathbf{z}, t_k, y) - \varepsilon_\theta(\mathbf{z}, t_k, \varnothing)\big)\) // classifier-free guidance
- \(\mathbf{z} \leftarrow \text{SolverStep}(\mathbf{z}, \hat\varepsilon, t_k, t_{k-1})\) // DDPM, DDIM or DPM-Solver step
- return \(\mathcal{D}(\mathbf{z} / s)\) // the decoder runs once, at the end
Algorithm 2 uses one solver step that you know from session 226. With guidance, the DDIM step from noise level \(t\) to the next, smaller level \(t'\) is
- Read the symbols
- \(w\)
- the guidance scale: \(w = 0\) ignores the condition, \(w = 1\) is the plain conditional model, \(w \gt 1\) pushes away from the unconditional prediction
- \(\hat{\mathbf{z}}_0\)
- the current guess of the clean latent, computed from the noise guess
- \(t'\)
- the next step of the schedule; at the last step \(\bar\alpha_{t'} = 1\), so \(\mathbf{z} = \hat{\mathbf{z}}_0\)
Figure 3 runs this loop on the toy latent space with the perfect denoiser (228.5). The steps are spread evenly from \(t = 999\) to \(t = 0\), as in the PyTorch code below. Watch three things: the weights \(w_k\) choose one training image as \(t\) falls, the preview \(\mathcal{D}(\hat{\mathbf{z}}_0 / s)\) sharpens, and the decoder runs only once, at the end.
- \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\), shape \(4 \times 4 \times 4\) // latent shape, no encoder
- for \(k = 1, \dots, K\) with \(t = t_k\):
- \(\varepsilon_c \leftarrow \varepsilon_\theta(\mathbf{z}, t, y)\), \(\varepsilon_u \leftarrow \varepsilon_\theta(\mathbf{z}, t, \varnothing)\)
- \(\hat\varepsilon \leftarrow \varepsilon_u + w(\varepsilon_c - \varepsilon_u)\) // (228.6)
- \(\hat{\mathbf{z}}_0 \leftarrow (\mathbf{z} - \sqrt{1 - \bar\alpha_t}\,\hat\varepsilon) / \sqrt{\bar\alpha_t}\)
- \(\mathbf{z} \leftarrow \sqrt{\bar\alpha_{t'}}\,\hat{\mathbf{z}}_0 + \sqrt{1 - \bar\alpha_{t'}}\,\hat\varepsilon\)
- \(\hat{\mathbf{x}} \leftarrow \mathcal{D}(\mathbf{z} / s)\) // the decoder runs once
How to read Figure 3
Defaults: class sun, \(w = 1\), \(K = 10\), seed 1. At the first step (\(t = 999\)) the unconditional weights are almost equal (0.253, 0.238, 0.254, 0.254), and the conditional weights split between the two sun images (0.515, 0.485). The first previews are blurry averages. A few steps later one image wins, and the final sample is closest to “sun left” (mean squared distance 0.0012, about the size of the data spread \(\sigma_d^2 = 0.01\) or less). Now switch to class sea with the same noise: the sample becomes “sea, low horizon” (0.0020). Set \(w = 0\): the condition is ignored, and this noise happens to land on “sea, low horizon” too. Try \(K = 5\): 5 loop steps (10 denoiser passes with guidance) still reach a clean sample here, because the perfect denoiser makes no errors for the solver to collect.
The first backbone: a U-Net with cross-attention
The original LDM and all Stable Diffusion 1.x/2.x/XL models use a U-Net as \(\varepsilon_\theta\). A U-Net is a convolutional encoder–decoder. The down path halves the resolution several times (for a 64×64 latent: 64, 32, 16, 8) while adding channels. The up path doubles it back. Skip connections copy features from each down level to the matching up level, so fine details do not have to pass through the small bottleneck. Each level has residual conv blocks; the noise step \(t\) enters every block as an added embedding. At the lower resolutions there are also attention layers.
The condition enters through cross-attention (session 20): the queries come from the image features, and the keys and values come from the text vectors \(\tau_\phi(y)\):
- Read the symbols
- \(\varphi\)
- the U-Net features at one level, flattened into a list of \(N\) spatial positions, shape \(N \times d_\varphi\)
- \(\tau = \tau_\phi(y)\)
- the text vectors, shape \(M \times d_\tau\) (77 tokens × 768 for CLIP ViT-L/14 in SD v1)
- \(W_Q, W_K, W_V\)
- learned projections into a shared width \(d\)
Each image position asks “which words are about me?” and pulls in their meaning. Stable Diffusion v1 has an 860M-parameter U-Net and a 123M-parameter CLIP ViT-L/14 text encoder, and works at \(f = 8\). [Certain] SDXL made the U-Net about three times larger, mostly through more attention blocks and a wider cross-attention context from a second text encoder. [Certain]
The U-Net has strong inductive biases: built-in assumptions about images. Convolutions are local and treat every position the same way, and the multi-scale design matches the fact that images have structure at many sizes. These biases help with little data. But they also make the network a hand-designed object: how many levels, where to put attention, how many channels at each level. Scaling it up is a craft.
The second backbone: the Diffusion Transformer (DiT)
Peebles & Xie asked a simple question: does the denoiser need the U-Net’s image-specific design at all? They took a standard Vision Transformer (session 164), changed as little as possible, and trained it on latents. The answer was no: with enough compute, the plain transformer is better, and its quality follows a clean scaling rule. [Certain]
Patchify: from latent to tokens
The input is the noisy latent, \(I \times I \times C\) with \(I = 32\) and \(C = 4\) for 256×256 images. DiT cuts it into non-overlapping \(p \times p\) patches, flattens each patch into a vector of \(p^2 C\) numbers, and maps it to width \(d\) with one linear layer. Fixed sine-cosine position embeddings are added. The number of tokens is
DiT tries \(p \in \{2, 4, 8\}\), giving \(T = 256, 64, 16\). Halving \(p\) multiplies \(T\) by 4 and the compute by at least 4, but leaves the parameter count almost unchanged: only the tiny input and output layers depend on \(p\). [Certain] So the patch size is a pure “compute knob”. Explore the order of the cut in Figure 4.
How to read Figure 4
Tokens are numbered row by row over the patch grid, like reading a page. With \(p = 1\), every latent position is a token of only 4 numbers: 64 tokens. With \(p = 8\), the whole latent is one token of 256 numbers, and attention has nothing to attend to. Unpatchify, at the output, is the exact inverse: the same reshape, run backwards.
The DiT block and how the condition gets in
A DiT block is a standard pre-norm transformer block: LayerNorm, self-attention, residual add, LayerNorm, MLP (width \(4d\), GELU), residual add (session 22). The question is how to feed in the step \(t\) and the class \(y\). First both become one vector per image:
DiT compared four ways of using \(\mathbf{c}\), all on DiT-XL/2 [Certain]:
- In-context: append \(\mathbf{c}\) as two extra tokens, like a [CLS] token. Almost free.
- Cross-attention: add a cross-attention layer to each block that attends to \(\mathbf{c}\) (as in the U-Net with text). Costs about 15% more compute.
- adaLN: replace the fixed LayerNorm gain and bias by values computed from \(\mathbf{c}\).
- adaLN-Zero: adaLN plus a learned gate on each residual branch, initialized to zero.
Adaptive LayerNorm (adaLN) is the same idea as FiLM: normalize the token, then scale and shift it with numbers that depend on the condition. A small network (SiLU then one linear layer) maps \(\mathbf{c}\) to a shift \(\beta\), a scale \(\gamma\), and a gate \(\alpha\), each of size \(d\), separately for the attention branch and the MLP branch (6 vectors per block):
- Read the symbols
- \(\text{LN}\)
- LayerNorm without its own learned gain and bias: subtract the mean, divide by the standard deviation
- \(\gamma, \beta\)
- per-channel scale and shift, computed from \(\mathbf{c}\); the same for all \(T\) tokens of one image
- \(\alpha\)
- per-channel gate on the residual branch; its producing layer starts at zero
- \(\odot\)
- element-wise multiplication
Why the zero? With \(\alpha = 0\) at the start, every block outputs its input unchanged: the whole network is the identity function. This trick is well known from ResNets (start each residual branch at zero) and makes large models train stably. DiT also zero-initializes the final output layer, so the first prediction is exactly zero. [Certain] Why adaLN instead of cross-attention? The condition is one vector per image, so attending to it adds little; scaling the normalized features is a cheaper and more direct way to say “we are at a high noise level, act accordingly”.
| Block design (DiT-XL/2, 400K steps) | Gflops | FID-50K, no guidance |
|---|---|---|
| In-context | 119.4 | 35.24 |
| Cross-attention | 137.6 | 26.14 |
| adaLN | 118.6 | 25.21 |
| adaLN-Zero | 118.6 | 19.47 |
Numbers from Peebles & Xie, Table 4. [Certain] adaLN-Zero is both the cheapest and the best. Figure 5 lets you move the scale, shift and gate on one toy token.
Take a token \(\mathbf{h} = (1, 2, 3, 4)\). LayerNorm: mean 2.5, variance 1.25, so \(\text{LN}(\mathbf{h}) = (-1.342, -0.447, 0.447, 1.342)\). Suppose the condition gives \(\gamma = 0.5\) and \(\beta = 0.1\) in every channel. Then formula (228.10) gives \(1.5 \cdot \text{LN}(\mathbf{h}) + 0.1 = (-1.912, -0.571, 0.771, 2.112)\). At a different step \(t\), \(\mathbf{c}\) changes, so \(\gamma\) and \(\beta\) change, and the same weights process the token differently. If the gate \(\alpha = 0\), the block output is \(\mathbf{h} = (1, 2, 3, 4)\) again, whatever the attention or MLP computes.
How to read Figure 5
With the gate at 0 (its value at initialization), the orange output bars sit exactly on the grey input bars: the block is the identity. Raise the gate and the branch starts to change the token. Move \(\gamma\) to \(-1\): the normalized token is multiplied by 0, so only \(\beta\) passes into the branch, which is how a condition can “switch off” the content of a token at some noise levels.
One whole DiT block, step by step
Figure 6 puts patchify, the condition vector and adaLN-Zero together in one DiT block, run on a noisy toy latent: \(\mathbf{z}_t\) of the “sun left” image from Figure 2, with fixed noise. The block is tiny (\(p = 2\), so \(T = 4\) tokens of \(p^2 C = 16\) numbers, width \(d = 8\), one attention head), and its weights are random numbers from a seeded generator, standing in for trained ones. So the output \(\hat\varepsilon\) is not a good noise prediction; the point is to watch the data flow and the conditioning. Two weight settings: “trained-like” uses random adaLN weights, and “adaLN-Zero init” sets the adaLN layer and the output layer to zero, as DiT does at the start of training.
- tokens \(\leftarrow\) patchify\((\mathbf{z}_t, p = 2)\) // (228.8): \(4 \times 16\)
- \(\mathbf{x} \leftarrow \text{tokens}\, W_e + \text{pos}\) // \(4 \times 8\)
- \(\mathbf{c} \leftarrow \text{MLP}(\text{sinusoid}(t)) + E[y]\) // (228.9)
- \((\beta_1, \gamma_1, \alpha_1, \beta_2, \gamma_2, \alpha_2) \leftarrow W_{\text{ada}}\, \text{SiLU}(\mathbf{c})\)
- \(\mathbf{h} \leftarrow \text{LN}(\mathbf{x}) \odot (1 + \gamma_1) + \beta_1\) // (228.10)
- \(\mathbf{x} \leftarrow \mathbf{x} + \alpha_1 \odot \text{Attn}(\mathbf{h})\) // (228.11)
- \(\mathbf{x} \leftarrow \mathbf{x} + \alpha_2 \odot \text{MLP}(\text{adaLN}_2(\mathbf{x}))\) // (228.11)
- \(\hat\varepsilon \leftarrow\) unpatchify\((\text{Linear}(\text{adaLN}_{\text{out}}(\mathbf{x})))\) // \(4 \times 4 \times 4\)
How to read Figure 6
Step to line 4 and press “Sweep t”: \(\mathbf{c}\), and with it \(\gamma_1\), \(\beta_1\) and \(\alpha_1\), change smoothly with the noise step. The same weights then process the tokens differently at each \(t\). At \(t = 500\) with class sun, the readout shows that the block changes the tokens by \(\lVert \Delta \mathbf{x} \rVert = 1.098\), and the largest output value is 3.403. Change the class to sea: \(\mathbf{c}\) changes (\(c[0]\) goes from −1.416 to −0.375) and the change becomes 0.729. Now choose “adaLN-Zero init”: lines 4 to 8 give \(\gamma = \beta = \alpha = 0\), the attention is still computed (look at the matrix), but nothing is added, so \(\lVert \Delta \mathbf{x} \rVert = 0\) and \(\hat\varepsilon = 0\) exactly, for every \(t\) and class. This is the identity start from the text.
The output layer
After the last block, a final adaLN and a linear layer map each token back to \(p \times p \times 2C\) numbers, and unpatchify puts them back into the latent grid. Why \(2C\) and not \(C\)? DiT predicts both the noise \(\varepsilon\) (\(C\) channels) and the variance of each reverse step (another \(C\) channels), as in Improved DDPM (Nichol & Dhariwal). The noise part is trained with the simple loss (228.4); the variance part with the full variational bound. [Certain] The Hugging Face config of DiT-XL-2-256 shows exactly this: in_channels: 4, out_channels: 8, patch_size: 2, num_layers: 28, 16 heads of size 72 (so \(d = 1152\)), and norm_type: ada_norm_zero. [Certain]
Counting the cost of a DiT
A DiT is a ViT, so we can count its cost with the same rules as an LLM (session 25). Per block, the attention projections have \(4d^2\) weights and the MLP \(8d^2\); every token uses each weight once. The attention scores and the weighted sum add \(2T^2 d\). DiT reports compute in “Gflops” that match multiply-add counts, so per forward pass:
The extra \(6d^2\) per block is the adaLN layer that maps \(\mathbf{c}\) to six vectors. It runs once per image, not per token, so it adds parameters but almost no compute.
DiT-XL has \(L = 28\), \(d = 1152\); with \(p = 2\), \(T = 256\). Per block: \(12 \cdot 256 \cdot 1152^2 = 4.077 \times 10^9\) for the weights and \(2 \cdot 256^2 \cdot 1152 = 0.151 \times 10^9\) for attention. Times 28 blocks: 118.4 Gflops. The paper reports 118.6 (the difference is the patch embedding, output layer and small terms). Parameters: \(18 \cdot 28 \cdot 1152^2 = 668.9\)M in the blocks; with the embeddings and final layer our count is 675.0M, and the paper says 675M (not counting the 84M VAE). [Certain]
At 512×512 the latent is 64×64, so \(T = 1024\) and the same model needs 524.6 Gflops (formula 228.12 gives 524.3). The pixel-space U-Net ADM needs 1120 Gflops at 256×256 and 1983 at 512×512. [Certain] Sampling with 250 DDPM steps costs \(250 \times 118.6 \approx 29.7\) Tflops per 256×256 image, and twice that with classifier-free guidance.
Scaling behavior of DiT
DiT trained 12 models: four sizes (S, B, L, XL) times three patch sizes (8, 4, 2), all for 400K steps at batch 256 with the same recipe (AdamW, constant learning rate \(10^{-4}\), no weight decay, EMA 0.9999). [Certain]
| Model | Layers \(L\) | Width \(d\) | Heads | Params (our count, 228.13 + extras) | Gflops at \(p = 4\) |
|---|---|---|---|---|---|
| DiT-S | 12 | 384 | 6 | 33M | 1.4 |
| DiT-B | 12 | 768 | 12 | 130M | 5.6 |
| DiT-L | 24 | 1024 | 16 | 458M | 19.7 |
| DiT-XL | 28 | 1152 | 16 | 675M | 29.1 |
Three findings matter [Certain]:
- Compute predicts quality. FID falls steadily as Gflops grow, and it does not matter much whether the Gflops come from a bigger model or from smaller patches (more tokens). The paper calls it a strong negative correlation between Gflops and FID.
- Parameters alone do not. DiT-XL/8 has about 675M parameters but only 7.39 Gflops, and its FID is 106.41. DiT-S/2 has about 20 times fewer parameters (33M) and 6.06 Gflops, and its FID is 68.40. Smaller patches, which give more tokens per image, matter more than parameters here.
- Bigger models use training compute better. At the same total training compute, larger DiTs reach lower FID. And extra sampling steps cannot make up for a small model: “scaling-up sampling compute cannot compensate for a lack of model compute.”
How to read Figure 7
Points line up along one falling line regardless of color: that is “compute predicts FID”. Compare points that sit at similar Gflops but come from different models, such as B/2 (23.0 Gflops, FID 43.47), L/4 (19.7, 45.64) and XL/4 (29.1, 43.01): very different shapes, similar quality. The readout also checks formula (228.12) against the paper for the selected model.
A least-squares line through the 12 points in log-log space has slope \(-0.365\) and correlation \(-0.94\). So, in this range, every 10× more forward compute divides FID by about \(10^{0.365} \approx 2.3\). This is our fit on the paper’s 12 numbers, not a law from the paper, and it holds only for this recipe and training length.
With long training (7M steps) and classifier-free guidance at scale 1.5, DiT-XL/2 reached FID 2.27 on ImageNet 256×256, better than LDM-4-G (3.60) and pixel-space ADM. At 512×512 it reached 3.04 after 3M steps. [Certain] Guidance trades diversity for quality, so the guided model has higher precision (0.83) and lower recall (0.57) than the unguided one (0.67 and 0.67). [Certain] More on this trade-off in session 227.
After DiT: text, video and joint models
DiT was class-conditional: the condition was one label. Text needs more than one vector, so later models put cross-attention back in. PixArt-\(\alpha\) adds a cross-attention layer to each DiT block for T5 text features and reports training at about 10.8% of the time of Stable Diffusion v1.5 (675 vs 6,250 A100 GPU-days). [Certain] Stable Diffusion 3 and FLUX go further: text tokens and image tokens are processed in one joint attention over both (MM-DiT), still with adaLN for the timestep, and they train with rectified flow instead of DDPM (session 229). [Likely] U-ViT (Bao et al., 2022) independently showed that a ViT with long skip connections works as a diffusion backbone. [Certain]
For video, the same recipe extends naturally. A video autoencoder compresses both space and time, and the latent is cut into spacetime patches (for example \(2 \times 2\) in space and a few frames in time), giving one long token sequence. OpenAI described Sora this way: a diffusion transformer on spacetime patches of a compressed video latent. [Likely] Because tokens are just a sequence, the same model can train on clips of different lengths, sizes and aspect ratios.
A minimal DiT in PyTorch
The code below builds a small DiT with patchify, adaLN-Zero blocks, the zero-initialized output, and trains it on toy “latents” (one fixed 4×8×8 pattern per class plus noise) with the DDPM loss. Then it samples with DDIM and classifier-free guidance, and checks that each sample is closest to the pattern of its class. In a real model, z0 would be s * vae.encode(x) and the samples would go through vae.decode(z / s). It runs on a CPU in about 15 seconds.
import math, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) def patchify(z, p): # (B, C, H, W) -> (B, T, p*p*C), formula (228.8) B, C, H, W = z.shape z = z.reshape(B, C, H // p, p, W // p, p).permute(0, 2, 4, 3, 5, 1) return z.reshape(B, (H // p) * (W // p), p * p * C) def unpatchify(x, p, C, H, W): # (B, T, p*p*C) -> (B, C, H, W) x = x.reshape(x.shape[0], H // p, W // p, p, p, C).permute(0, 5, 1, 3, 2, 4) return x.reshape(x.shape[0], C, H, W) def t_embed(t, dim=256): # sinusoidal timestep features half = dim // 2 freqs = torch.exp(-math.log(10000) * torch.arange(half) / half) a = t[:, None].float() * freqs[None] return torch.cat([torch.cos(a), torch.sin(a)], dim=-1) def modulate(x, shift, scale): # formula (228.10) return x * (1 + scale[:, None]) + shift[:, None] class DiTBlock(nn.Module): def __init__(self, d, heads): super().__init__() self.n1 = nn.LayerNorm(d, elementwise_affine=False, eps=1e-6) self.n2 = nn.LayerNorm(d, elementwise_affine=False, eps=1e-6) self.attn = nn.MultiheadAttention(d, heads, batch_first=True) self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(approximate="tanh"), nn.Linear(4 * d, d)) self.ada = nn.Sequential(nn.SiLU(), nn.Linear(d, 6 * d)) nn.init.zeros_(self.ada[1].weight); nn.init.zeros_(self.ada[1].bias) # adaLN-Zero def forward(self, x, c): # formula (228.11) b1, g1, a1, b2, g2, a2 = self.ada(c).chunk(6, dim=-1) h = modulate(self.n1(x), b1, g1) x = x + a1[:, None] * self.attn(h, h, h, need_weights=False)[0] x = x + a2[:, None] * self.mlp(modulate(self.n2(x), b2, g2)) return x class TinyDiT(nn.Module): def __init__(self, I=8, C=4, p=2, d=64, depth=4, heads=4, n_cls=10): super().__init__() self.I, self.C, self.p = I, C, p T = (I // p) ** 2 self.x_emb = nn.Linear(p * p * C, d) self.pos = nn.Parameter(0.02 * torch.randn(1, T, d)) self.t_mlp = nn.Sequential(nn.Linear(256, d), nn.SiLU(), nn.Linear(d, d)) self.y_emb = nn.Embedding(n_cls + 1, d) # last id = "no class" (for CFG) self.blocks = nn.ModuleList([DiTBlock(d, heads) for _ in range(depth)]) self.n_out = nn.LayerNorm(d, elementwise_affine=False, eps=1e-6) self.ada_out = nn.Sequential(nn.SiLU(), nn.Linear(d, 2 * d)) self.out = nn.Linear(d, p * p * C) # real DiT: p*p*2C (noise + variance) for m in (self.ada_out[1], self.out): # zero init: output starts at 0 nn.init.zeros_(m.weight); nn.init.zeros_(m.bias) def forward(self, z, t, y): x = self.x_emb(patchify(z, self.p)) + self.pos # (B, T, d) c = self.t_mlp(t_embed(t)) + self.y_emb(y) # (B, d), formula (228.9) for blk in self.blocks: x = blk(x, c) b, g = self.ada_out(c).chunk(2, dim=-1) x = self.out(modulate(self.n_out(x), b, g)) # (B, T, p*p*C) return unpatchify(x, self.p, self.C, self.I, self.I) # predicted noise # DDPM schedule as in DiT: 1000 steps, linear betas from 1e-4 to 2e-2 betas = torch.linspace(1e-4, 2e-2, 1000) abar = torch.cumprod(1 - betas, dim=0) model = TinyDiT() print("params:", sum(q.numel() for q in model.parameters())) # 331536 z = torch.randn(2, 4, 8, 8) print("round trip:", torch.equal(unpatchify(patchify(z, 2), 2, 4, 8, 8), z)) # True print("output at init:", model(z, torch.tensor([10, 500]), torch.tensor([3, 10])).abs().max().item()) # 0.0 # toy "latents": each of 10 classes is a fixed 4x8x8 pattern plus a little noise protos = torch.randn(10, 4, 8, 8) opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.0) for step in range(1001): y = torch.randint(0, 10, (64,)) z0 = protos[y] + 0.1 * torch.randn(64, 4, 8, 8) # real LDM: z0 = s * vae.encode(x) y_in = torch.where(torch.rand(64) < 0.1, torch.full_like(y, 10), y) # drop the label 10% of the time t = torch.randint(0, 1000, (64,)) eps = torch.randn_like(z0) a = abar[t].view(-1, 1, 1, 1) zt = a.sqrt() * z0 + (1 - a).sqrt() * eps # forward noising, in latent space loss = F.mse_loss(model(zt, t, y_in), eps) # formula (228.4) opt.zero_grad(); loss.backward(); opt.step() if step % 250 == 0: print(step, round(loss.item(), 4)) # 0.9967 at step 0, below 0.1 by step 1000 # sampling: deterministic DDIM, 50 steps, classifier-free guidance w = 2 @torch.no_grad() def sample(y, w=2.0, steps=50): ts = torch.linspace(999, 0, steps).long() zt = torch.randn(len(y), 4, 8, 8) for i, t in enumerate(ts): tt = t.repeat(len(y)) e_c = model(zt, tt, y) e_u = model(zt, tt, torch.full_like(y, 10)) e = e_u + w * (e_c - e_u) a = abar[t] z0_hat = (zt - (1 - a).sqrt() * e) / a.sqrt() a_prev = abar[ts[i + 1]] if i + 1 < steps else torch.tensor(1.0) zt = a_prev.sqrt() * z0_hat + (1 - a_prev).sqrt() * e return zt # real LDM: x = vae.decode(zt / s) y = torch.arange(10) zs = sample(y) dist = ((zs[:, None] - protos[None]) ** 2).mean(dim=(2, 3, 4)) # (10 samples, 10 patterns) print("nearest pattern:", dist.argmin(1).tolist()) # [0, 1, 2, ..., 9] print("error to own pattern:", round(dist.diag().mean().item(), 4)) # about 0.018 (data noise alone: 0.01)
Three checks in the output: the patchify round trip is exact, the untrained model outputs exactly 0 (adaLN-Zero plus the zero output layer), and every one of the 10 samples lands nearest to the pattern of the class it was asked for. The loss falls from 0.9967 at step 0 to below 0.1 by step 1,000 (CPU run, PyTorch 2.x; exact later values can differ slightly on other hardware).
Where this is used
| System or task | Latent space | Denoiser | Notes |
|---|---|---|---|
| Stable Diffusion 1.x (text-to-image) | \(f = 8\), \(c = 4\) KL autoencoder | 860M U-Net, cross-attention to CLIP text | The model that made LDM popular; runs on a consumer GPU [Certain] |
| SDXL | \(f = 8\), \(c = 4\), \(s = 0.13025\) | About 3× larger U-Net, two text encoders | [Certain] |
| DiT (class-conditional ImageNet) | Stable Diffusion VAE, 32×32×4 for 256×256 | Transformer with adaLN-Zero | FID 2.27 with guidance [Certain] |
| PixArt-\(\alpha\), SD3, FLUX (text-to-image) | \(f = 8\); SD3 and FLUX use 16 channels [Likely] | DiT with cross-attention (PixArt) or joint text-image attention (MM-DiT) | Next session for the training objective |
| Text-to-video generation (Sora and similar) | Video autoencoder that compresses space and time | DiT on spacetime patches | Variable length and aspect ratio come almost for free with tokens [Likely] |
| Video editing, inpainting and super-resolution | Same latent; known regions are re-noised and kept | U-Net or DiT with extra input channels (mask, low-res frame) | LDM already did inpainting and super-resolution this way [Certain] |
| Synthetic training data for video tasks (action recognition, key-moment detection) | Any latent video model | Any | Generate rare actions or camera views to balance a dataset; check for a domain gap [Speculation] |
| Unified VLMs that also draw (e.g. Transfusion) | VAE latent patches inside the LLM sequence | The LLM transformer itself, with a diffusion loss on image patches | [Certain]; see session 232 |
Common mistakes
- Forgetting the latent scale \(s\). Training on raw VAE latents (std far from 1) silently changes every noise level of the schedule. Multiply by \(s\) after encoding and divide by \(s\) before decoding, exactly once each.
- Mixing VAEs. A denoiser trained on one autoencoder’s latents is useless with another autoencoder (for example the 4-channel SD VAE vs a 16-channel VAE), even if the shapes match.
- Thinking the latent is a nice Gaussian. With KL weight around \(10^{-6}\), the latent is closer to a plain autoencoder code. Interpolating or sampling it directly does not give images; only the diffusion model can.
- Choosing \(f\) too large. Details lost by the encoder (small text, faces far away) cannot be recovered by the denoiser. When a generator draws bad small text, check the autoencoder’s reconstruction first.
- Comparing DiTs by parameter count. Patch size changes compute 4× per halving with almost no change in parameters; compare by Gflops (formula 228.12).
- Using a normal random init for the adaLN layers. Without the zero init, deep DiTs start far from the identity and train worse; the paper’s FID gap between adaLN and adaLN-Zero is 25.21 vs 19.47.
- Wrong patch order. Patchify and unpatchify must be exact inverses. A mismatched reshape still trains, but the model must learn to undo the scramble, and position embeddings no longer match the content.
- Running the VAE in FP16 when it overflows. Some VAEs (the SDXL VAE config sets
force_upcast: true) produce NaNs in half precision; decode in FP32 or use a fixed VAE. [Likely]
Interview questions
Why do latent diffusion models diffuse in a latent space instead of pixels?basic
Walk me through the shapes in Stable Diffusion v1 for a 512×512 image.basic
What does the scale factor 0.18215 do, and what happens if you forget it?core
How does DiT turn a latent into tokens, and what does the patch size control?core
Explain adaLN-Zero. Why is it better than cross-attention for class conditioning in DiT?core
What is the scaling result of the DiT paper?core
Estimate the forward cost of DiT-XL/2 from its config.advanced
Why does DiT output 2C channels instead of C?advanced
U-Net vs DiT: what are the trade-offs?core
How would you condition a DiT on a text prompt instead of a class?advanced
Your latent diffusion model draws blurry small text and faces. Where do you look first?advanced
Exercises
- Compute the latent shape and the compression ratio for a 768×768 image with \(f = 8\), \(c = 4\), and with \(f = 8\), \(c = 16\).
- A DiT-B (\(L = 12\), \(d = 768\)) runs on a 32×32×4 latent with \(p = 4\). How many tokens? Estimate the Gflops with formula (228.12) and compare with Figure 7.
- Show that if \(\alpha = 0\) in every block and the output layer is zero, the DiT output is 0 for any input. Why does this not stop learning?
- You want 256×256 images with a DiT-L but can only afford about 20 Gflops per forward. Which patch size? What FID does Table 4 report at 400K steps?
- A latent has standard deviation 5.5 instead of 1. At a step where \(\bar\alpha_t = 0.5\), what is the signal-to-noise ratio \(\bar\alpha_t \sigma_z^2 / (1 - \bar\alpha_t)\), compared to the intended value?
- Count the tokens of a video DiT for 16 frames of 256×256 with a video autoencoder that compresses 8× in space and 4× in time, and patches of \(2 \times 2\) in space and 1 in time.
Answerscheck
Key takeaways
- Latent diffusion splits generation into perceptual compression (an autoencoder, trained once) and semantic generation (diffusion on the small latent), formula (228.1). Stable Diffusion: \(f = 8\), \(c = 4\), 48× fewer values (228.2).
- The diffusion math is unchanged; only the data is \(\mathbf{z}_0 = s\,\mathcal{E}(\mathbf{x})\) (228.4). The scale \(s\) (0.18215 for SD 1.x) keeps the noise schedule valid.
- The U-Net backbone uses convolutions, skips and cross-attention (228.7); DiT replaces it with a plain ViT on latent patches (228.8).
- DiT conditions through adaLN-Zero (228.10, 228.11): scale, shift and gate computed from the timestep and class; zero init makes every block start as the identity.
- DiT quality follows forward compute (228.12), not parameter count; smaller patches and bigger models both help, and bigger models use training compute better. DiT-XL/2: 675M params, 118.6 Gflops, FID 2.27 with guidance.
Key formulas
References for this session
- Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models
- Peebles & Xie (2022). Scalable Diffusion Models with Transformers
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 10: Any-to-Any Models
- Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models
- Peebles & Xie (2022). Scalable Diffusion Models with Transformers
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 10: Any-to-Any Models
- Ho et al. (2020). Denoising Diffusion Probabilistic Models
- Song et al. (2020). Denoising Diffusion Implicit Models
- Ho & Salimans (2022). Classifier-Free Diffusion Guidance
- Dhariwal & Nichol (2021). Diffusion Models Beat GANs on Image Synthesis
- Esser et al. (2020). Taming Transformers for High-Resolution Image Synthesis
- Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Ronneberger, Fischer & Brox (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation
- Nichol & Dhariwal (2021). Improved Denoising Diffusion Probabilistic Models
- Perez et al. (2017). FiLM: Visual Reasoning with a General Conditioning Layer
- Bao et al. (2022). All are Worth Words: A ViT Backbone for Diffusion Models
- Chen et al. (2023). PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis
- Podell et al. (2023). SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
- Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- Brooks et al. (OpenAI) (2024). Video generation models as world simulators (technical report)
- Zhou et al. (2024). Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model