LLM & VLM courseBlog
228Track 9: Generative models · session 7 of 12

Latent diffusion and diffusion transformers (DiT)

Modern image and video generators do not run diffusion on pixels. They first compress the image with an autoencoder and run diffusion on the small latent, then decode once at the end. This session builds latent diffusion step by step, compares the U-Net backbone with the Diffusion Transformer (DiT), explains patchify and adaLN-Zero conditioning, and reads the DiT scaling results with real numbers.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Diffusion 1 and 2 (sessions 225 and 226), variational autoencoders (session 223), the Vision Transformer (session 164); basic PyTorch

What you will learn

  • Why diffusing in a compressed latent space is cheaper and better: perceptual vs semantic compression, and how to count the 48× saving of the Stable Diffusion autoencoder
  • How the autoencoder is trained (pixel, perceptual, adversarial and tiny KL losses), why the downsampling factor f = 4 to 8 works best, and why the latent scale 0.18215 matters
  • The latent diffusion training and sampling algorithms, and where the encoder, denoiser and decoder each run
  • The U-Net backbone with cross-attention to text, and its inductive biases
  • The Diffusion Transformer: patchify, the token count T = (I/p)², adaLN and adaLN-Zero conditioning, and the 2C-channel output
  • How to estimate the Gflops and parameters of a DiT from its config, and the DiT scaling result: compute, not parameter count, predicts FID
  • How DiT led to text-to-image and video transformers (PixArt-α, MM-DiT, spacetime patches)

Why this matters

Almost every well-known image and video generator since 2022 does its diffusion in a small latent space, not on pixels. Stable Diffusion, SDXL, PixArt, Stable Diffusion 3, FLUX and Sora all follow this plan. And since 2023, most new ones also replaced the convolutional U-Net with a plain transformer, the Diffusion Transformer (DiT).

In session 225 and session 226 we learned to train a network that predicts noise, and to sample by removing noise step by step. Two problems remain. First, cost: a 512×512 RGB image has 786,432 numbers, and the network runs on all of them at every one of tens or hundreds of sampling steps. Second, most of those numbers are tiny details (exact texture, sensor noise) that the eye barely notices but that a pixel-space model must still spend capacity on. This session fixes both. We first compress the image with an autoencoder, then run diffusion on the small code. Then we change the denoiser from a U-Net to a transformer, which brings the scaling rules we know from LLMs to image generation.

The two papers behind this session are Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models (LDM, 2021), and Peebles & Xie, Scalable Diffusion Models with Transformers (DiT, 2022). We will read their key tables with real numbers.

Step 1: compress first, then diffuse

Think about what a generator must learn. Rombach et al. split it into two kinds of work. Perceptual compression removes high-frequency details that are hard to see but expensive to model. Semantic compression is the real creative part: which objects are in the scene, where, and how they relate. A pixel-space diffusion model does both jobs with one network. LDM gives the first job to an autoencoder that is trained once, and the second job to the diffusion model. [Certain]

The autoencoder has an encoder \(\mathcal{E}\) and a decoder \(\mathcal{D}\). The encoder shrinks the height and width by a downsampling factor \(f\) and outputs \(c\) channels:

\[ \mathbf{z}_0 = s \cdot \mathcal{E}(\mathbf{x}) \in \R^{\frac{H}{f} \times \frac{W}{f} \times c}, \qquad \hat{\mathbf{x}} = \mathcal{D}(\mathbf{z}_0 / s) \in \R^{H \times W \times 3} \tag{228.1} \]
Read the symbols
\(\mathbf{x}\)
the image, \(H \times W\) pixels with 3 colors, scaled to \([-1, 1]\)
\(\mathcal{E}, \mathcal{D}\)
the encoder and decoder of the autoencoder; both are convolutional networks, frozen during diffusion training
\(f\)
the downsampling factor: each latent position covers an \(f \times f\) block of pixels
\(c\)
the number of latent channels (4 in Stable Diffusion 1.x and 2.x and SDXL)
\(s\)
a fixed scale factor that makes the latent values have standard deviation near 1 (explained below)
\(\mathbf{z}_0\)
the clean latent: the “data” for the diffusion model

The number of values shrinks by

\[ \text{compression} = \frac{H \cdot W \cdot 3}{\frac{H}{f} \cdot \frac{W}{f} \cdot c} = \frac{3 f^2}{c} \tag{228.2} \]
Worked example: the Stable Diffusion autoencoder

Stable Diffusion v1 uses \(f = 8\) and \(c = 4\). [Certain] A 512×512 image has \(512 \cdot 512 \cdot 3 = 786{,}432\) values. Its latent is \(64 \times 64 \times 4 = 16{,}384\) values. The ratio is \(3 \cdot 8^2 / 4 = 48\). For the 256×256 images used by DiT, the latent is \(32 \times 32 \times 4 = 4{,}096\) values instead of \(196{,}608\). Again 48 times fewer.

Note that the spatial grid shrinks by \(f^2 = 64\), but the channel count grows from 3 to 4. So the latent is not 64 times smaller, “only” 48. Newer autoencoders (SD3, FLUX) keep \(f = 8\) but use \(c = 16\) to keep more detail, which gives \(3 \cdot 64/16 = 12\). [Likely]

How the autoencoder is trained

The autoencoder is a VAE-like network (session 223) but tuned for sharp reconstructions, much like VQGAN (session 224). Its loss mixes a pixel loss, a perceptual loss (distance between deep features of \(\mathbf{x}\) and \(\hat{\mathbf{x}}\)), a patch-based adversarial loss (a small discriminator that judges local realism), and a tiny regularizer on the latent:

\[ \mathcal{L}_{\text{AE}} = \lVert \mathbf{x} - \hat{\mathbf{x}} \rVert_1 + \lambda_{P}\, \mathcal{L}_{\text{perc}}(\mathbf{x}, \hat{\mathbf{x}}) + \lambda_{G}\, \mathcal{L}_{\text{GAN}}(\hat{\mathbf{x}}) + \lambda_{\text{KL}}\, \KL\big(q(\mathbf{z} \mid \mathbf{x}) \,\Vert\, \mathcal{N}(\mathbf{0}, I)\big) \tag{228.3} \]

The KL weight is very small (about \(10^{-6}\) in the released configs [Likely]). The KL term only stops the latent from drifting to huge values; it does not force the latent to be a nice Gaussian like in a real VAE. LDM also has a second option, a VQ layer instead of the KL term (“VQ-reg”). [Certain]

The scale factor \(s\)

Because the KL term is so weak, the raw latent has some arbitrary spread. The noise schedule from session 225 assumes data with a standard deviation near 1: at step \(t\), the signal-to-noise ratio is \(\bar\alpha_t / (1 - \bar\alpha_t)\) times the data variance. If the latent is 5 times too large, every noise level is effectively 25 times weaker than planned. So LDM measures the standard deviation of the latents on a batch of images and multiplies by \(s = 1/\text{std}\). For the Stable Diffusion 1.x VAE, \(s = 0.18215\); for the SDXL VAE, \(s = 0.13025\). [Certain] Forgetting \(s\), or dividing by it twice at decode time, is a classic bug.

How much to compress: the factor \(f\)

LDM tried \(f \in \{1, 2, 4, 8, 16, 32\}\) on ImageNet. With \(f = 1\) or 2 (almost pixels), training is slow. With \(f = 32\), quality stops improving early: the autoencoder has thrown away detail that no diffusion model can bring back. \(f = 4\) to 16 is the good range, with \(f = 4\) and 8 best overall. After 2 million steps, LDM-8 had an FID about 38 points better than the pixel-space LDM-1. [Certain] FID (Fréchet Inception Distance) compares statistics of deep features of real and generated images; lower is better.

Figure 1 lets you play with \(H\), \(f\), \(c\) and the patch size \(p\) that the transformer will use later.

Image size Factor \(f\) Channels \(c\) Patch \(p\)
Figure 1. From pixels to latents to transformer tokens. Left: the image. Middle: the latent grid, \(H/f\) on each side, with \(c\) channels stacked behind it. Right: the token grid after cutting the latent into \(p \times p\) patches. The readout counts values, tokens and the size of one attention matrix.
How to read Figure 1

Start at the DiT setting (256, \(f = 8\), \(c = 4\), \(p = 2\)): 48× compression and 256 tokens. Now set \(f = 1\): the same transformer on raw pixels would need 16,384 tokens, and each attention matrix would be 4,096 times larger. Set \(c = 16\) to see how modern VAEs trade compression for detail. Try 1024 with \(f = 8\), \(p = 2\): 4,096 tokens, which is why high-resolution DiTs are expensive.

Step 2: diffusion in the latent space

Once the autoencoder is frozen, latent diffusion is exactly the DDPM of session 225, with \(\mathbf{x}_0\) replaced by \(\mathbf{z}_0\). We add noise, \(\mathbf{z}_t = \sqrt{\bar\alpha_t}\,\mathbf{z}_0 + \sqrt{1 - \bar\alpha_t}\,\varepsilon\), and train the denoiser to predict \(\varepsilon\):

\[ \mathcal{L}_{\text{LDM}} = \E_{\mathbf{x},\, y,\, \varepsilon \sim \mathcal{N}(\mathbf{0}, I),\, t} \Big[ \big\lVert \varepsilon - \varepsilon_\theta\big(\mathbf{z}_t,\, t,\, \tau_\phi(y)\big) \big\rVert_2^2 \Big], \qquad \mathbf{z}_0 = s\,\mathcal{E}(\mathbf{x}) \tag{228.4} \]
Read the symbols
\(\varepsilon_\theta\)
the denoiser (a U-Net or a DiT); it sees the noisy latent, the step and the condition
\(y\)
the condition: a class label, a text prompt, a depth map, ...
\(\tau_\phi\)
the condition encoder, for example a frozen CLIP text encoder that turns the prompt into a sequence of vectors
\(t\)
the noise step, drawn uniformly from \(1, \dots, 1000\)

Nothing about the diffusion math changes. That is the beauty of the design: the autoencoder is a plug-in. You can train it once and reuse it for many diffusion models. DiT, for example, did not train its own autoencoder at all; it used the off-the-shelf Stable Diffusion VAE. [Certain] Sampling also stays the same (DDPM, DDIM, or the fast solvers of session 226), and only at the very end do we call the decoder once.

Algorithm 1: one training step of latent diffusion
  1. input: a batch of images \(\mathbf{x}\) and conditions \(y\); frozen \(\mathcal{E}\), scale \(s\); denoiser \(\varepsilon_\theta\); schedule \(\bar\alpha_1, \dots, \bar\alpha_{1000}\)
  2. \(\mathbf{z}_0 \leftarrow s \cdot \mathcal{E}(\mathbf{x})\) // no gradient; often pre-computed once for the whole dataset
  3. with probability 0.1, replace \(y\) by the empty condition \(\varnothing\) // enables classifier-free guidance
  4. \(t \sim \text{Uniform}\{1, \dots, 1000\}\), \(\;\varepsilon \sim \mathcal{N}(\mathbf{0}, I)\)
  5. \(\mathbf{z}_t \leftarrow \sqrt{\bar\alpha_t}\,\mathbf{z}_0 + \sqrt{1 - \bar\alpha_t}\,\varepsilon\)
  6. \(\mathcal{L} \leftarrow \lVert \varepsilon - \varepsilon_\theta(\mathbf{z}_t, t, \tau_\phi(y)) \rVert^2\) // formula (228.4)
  7. update \(\theta\) with AdamW; update an EMA copy of \(\theta\) // sample with the EMA weights

Watching the training step on a toy latent space

To see Algorithm 1 run with every number visible, we need a toy world. Our “images” are 16×16 RGB pictures in two classes. Sun: a yellow disc on a blue sky, on the left or on the right. Sea: light sky over dark water, with the horizon high or low. The toy encoder uses \(f = 4\) and \(c = 4\): for each 4×4 block of pixels it keeps the mean red, green and blue, plus one “edge” number that says how much the block gets brighter or darker from top to bottom, all multiplied by 5 (a real encoder also has some arbitrary output scale). So 768 pixel values become 64 latent values: \(3 \cdot 4^2 / 4 = 12\) times fewer, formula (228.2). The decoder paints each block back from its four numbers.

The round edge of the sun does not survive this encoder: the reconstruction error (mean squared, pixels in \([-1, 1]\)) is 0.0845 for the sun images and only 0.0069 for the sea images, whose horizon is a straight line that the edge channel can describe. That is the “\(f\) too large” problem in miniature. The raw latents have standard deviation 2.305, so \(s = 1/2.305 = 0.434\).

We also need a denoiser. Instead of training a network, we use the perfect one. When the data is a small set of known latents \(\mu_1, \dots, \mu_4\), each with a little Gaussian spread \(\sigma_d\) (we use 0.1), the noise prediction with the lowest possible loss (228.4) can be written down exactly. It first guesses which training latent \(\mathbf{z}_t\) came from, then removes the noise:

\[ \begin{aligned} w_k &= \softmax_k\!\Big( -\frac{\lVert \mathbf{z}_t - \sqrt{\bar\alpha_t}\,\mu_k \rVert^2}{2 v_t} \Big), \qquad v_t = \bar\alpha_t \sigma_d^2 + 1 - \bar\alpha_t \\ \hat{\mathbf{z}}_0 &= \sum_{k} w_k \Big( \mu_k + \frac{\sqrt{\bar\alpha_t}\, \sigma_d^2}{v_t} \big(\mathbf{z}_t - \sqrt{\bar\alpha_t}\,\mu_k\big) \Big) \\ \varepsilon^\ast(\mathbf{z}_t, t, y) &= \frac{\mathbf{z}_t - \sqrt{\bar\alpha_t}\, \hat{\mathbf{z}}_0}{\sqrt{1 - \bar\alpha_t}} \end{aligned} \tag{228.5} \]
Read the symbols
\(\mu_k\)
the scaled latent of training image \(k\); the sum runs over the images of class \(y\), or over all four when \(y = \varnothing\)
\(w_k\)
the probability that \(\mathbf{z}_t\) came from image \(k\): how well \(\mathbf{z}_t\) fits \(\sqrt{\bar\alpha_t}\,\mu_k\) plus noise of variance \(v_t\)
\(\hat{\mathbf{z}}_0\)
the best guess of the clean latent, \(\E[\mathbf{z}_0 \mid \mathbf{z}_t, y]\)
\(\varepsilon^\ast\)
the best guess of the noise; a trained \(\varepsilon_\theta\) tries to get as close as possible to this function

A real \(\varepsilon_\theta\) never sees this formula: it must learn the same mapping from millions of examples, for data far too large to list. But using the perfect one lets us watch the algorithm itself, with the learning problem already solved. Figure 2 runs Algorithm 1 on 8 random training examples. Every number is computed in your browser with the 1000-step linear schedule of DiT.

Random seedScale
  1. \(\mathbf{z}_0 \leftarrow s \cdot \mathcal{E}(\mathbf{x})\) // 768 values \(\to\) 64
  2. with probability 0.1: \(y \leftarrow \varnothing\)
  3. \(t \sim \text{Uniform}\{0, \dots, 999\}\), \(\varepsilon \sim \mathcal{N}(\mathbf{0}, I)\)
  4. \(\mathbf{z}_t \leftarrow \sqrt{\bar\alpha_t}\,\mathbf{z}_0 + \sqrt{1 - \bar\alpha_t}\,\varepsilon\)
  5. \(\hat\varepsilon \leftarrow \varepsilon_\theta(\mathbf{z}_t, t, y)\), \(\mathcal{L} \leftarrow \operatorname{mean}(\varepsilon - \hat\varepsilon)^2\) // (228.4), with (228.5) as \(\varepsilon_\theta\)
example with its labellabel dropped (\(y = \varnothing\))
Figure 2. Algorithm 1 on the toy latent space, one line per step. Panels: the image, its latent \(\mathbf{z}_0\) (channels R, G, B and edge; blue is positive, orange negative), the noise \(\varepsilon\), the noisy latent \(\mathbf{z}_t\), the prediction \(\hat\varepsilon\) of the perfect denoiser (228.5), and \(\mathcal{D}(\mathbf{z}_t / s)\), the noisy latent decoded only so you can see it (training never decodes). The live box shows each formula with the numbers of the first latent entry, \(\mathbf{z}[R, 0, 0]\). The plot collects one dot per example.
How to read Figure 2

With seed 1 the first example is “sea, high horizon”. Its coin is \(u = 0.003 \lt 0.1\), so the label is dropped. It gets \(t = 527\), where \(\bar\alpha_t = 0.0588\): the decoded \(\mathbf{z}_t\) is mostly noise, and the signal-to-noise ratio is \(0.0588/0.9412 = 0.062\). The loss is 0.0021. Most dots sit near 0: the denoiser is perfect. Now pick seed 3 and step to example 5: \(t = 33\) and the loss is 0.1375, the largest one. At small \(t\) the noise is so small that it hides inside the natural spread \(\sigma_d\) of the data, and nobody can tell them apart. Finally choose “forget s”: the latent has standard deviation 2.305 instead of 1, and at the same \(t = 527\) the ratio becomes 0.332, \(2.305^2 = 5.3\) times more signal than the schedule planned. This is the bug from the scale section, now with numbers.

Algorithm 2: sampling an image with latent diffusion and guidance
  1. input: condition \(y\), guidance scale \(w\), number of steps \(K\), decoder \(\mathcal{D}\), scale \(s\)
  2. \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\) with the latent shape, e.g. \(4 \times 64 \times 64\) for a 512×512 image
  3. for \(k = K, \dots, 1\) (steps \(t_K \gt \dots \gt t_1\)):
  4.   \(\hat\varepsilon \leftarrow \varepsilon_\theta(\mathbf{z}, t_k, \varnothing) + w\,\big(\varepsilon_\theta(\mathbf{z}, t_k, y) - \varepsilon_\theta(\mathbf{z}, t_k, \varnothing)\big)\) // classifier-free guidance
  5.   \(\mathbf{z} \leftarrow \text{SolverStep}(\mathbf{z}, \hat\varepsilon, t_k, t_{k-1})\) // DDPM, DDIM or DPM-Solver step
  6. return \(\mathcal{D}(\mathbf{z} / s)\) // the decoder runs once, at the end

Algorithm 2 uses one solver step that you know from session 226. With guidance, the DDIM step from noise level \(t\) to the next, smaller level \(t'\) is

\[ \begin{aligned} \hat\varepsilon &= \varepsilon_\theta(\mathbf{z}, t, \varnothing) + w \big( \varepsilon_\theta(\mathbf{z}, t, y) - \varepsilon_\theta(\mathbf{z}, t, \varnothing) \big) \\ \hat{\mathbf{z}}_0 &= \frac{\mathbf{z} - \sqrt{1 - \bar\alpha_t}\, \hat\varepsilon}{\sqrt{\bar\alpha_t}}, \qquad \mathbf{z} \leftarrow \sqrt{\bar\alpha_{t'}}\, \hat{\mathbf{z}}_0 + \sqrt{1 - \bar\alpha_{t'}}\, \hat\varepsilon \end{aligned} \tag{228.6} \]
Read the symbols
\(w\)
the guidance scale: \(w = 0\) ignores the condition, \(w = 1\) is the plain conditional model, \(w \gt 1\) pushes away from the unconditional prediction
\(\hat{\mathbf{z}}_0\)
the current guess of the clean latent, computed from the noise guess
\(t'\)
the next step of the schedule; at the last step \(\bar\alpha_{t'} = 1\), so \(\mathbf{z} = \hat{\mathbf{z}}_0\)

Figure 3 runs this loop on the toy latent space with the perfect denoiser (228.5). The steps are spread evenly from \(t = 999\) to \(t = 0\), as in the PyTorch code below. Watch three things: the weights \(w_k\) choose one training image as \(t\) falls, the preview \(\mathcal{D}(\hat{\mathbf{z}}_0 / s)\) sharpens, and the decoder runs only once, at the end.

Class \(y\)Guidance \(w\)Steps \(K\)
Noise seedStep by
  1. \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\), shape \(4 \times 4 \times 4\) // latent shape, no encoder
  2. for \(k = 1, \dots, K\) with \(t = t_k\):
  3. \(\varepsilon_c \leftarrow \varepsilon_\theta(\mathbf{z}, t, y)\), \(\varepsilon_u \leftarrow \varepsilon_\theta(\mathbf{z}, t, \varnothing)\)
  4. \(\hat\varepsilon \leftarrow \varepsilon_u + w(\varepsilon_c - \varepsilon_u)\) // (228.6)
  5. \(\hat{\mathbf{z}}_0 \leftarrow (\mathbf{z} - \sqrt{1 - \bar\alpha_t}\,\hat\varepsilon) / \sqrt{\bar\alpha_t}\)
  6. \(\mathbf{z} \leftarrow \sqrt{\bar\alpha_{t'}}\,\hat{\mathbf{z}}_0 + \sqrt{1 - \bar\alpha_{t'}}\,\hat\varepsilon\)
  7. \(\hat{\mathbf{x}} \leftarrow \mathcal{D}(\mathbf{z} / s)\) // the decoder runs once
Figure 3. Algorithm 2 (DDIM with classifier-free guidance) on the toy latent space. Panels: the latent \(\mathbf{z}\), the guided noise \(\hat\varepsilon\), the decoded guess \(\mathcal{D}(\hat{\mathbf{z}}_0 / s)\), the weights \(w_k\) of (228.5) over the four training images (green: images of the chosen class), and the final image. The strip shows the decoded guess after every step, labelled with \(t\). The live box evaluates (228.6) on the entry \(\mathbf{z}[R, 0, 0]\). The previews are for teaching: a real sampler decodes only at the end.
How to read Figure 3

Defaults: class sun, \(w = 1\), \(K = 10\), seed 1. At the first step (\(t = 999\)) the unconditional weights are almost equal (0.253, 0.238, 0.254, 0.254), and the conditional weights split between the two sun images (0.515, 0.485). The first previews are blurry averages. A few steps later one image wins, and the final sample is closest to “sun left” (mean squared distance 0.0012, about the size of the data spread \(\sigma_d^2 = 0.01\) or less). Now switch to class sea with the same noise: the sample becomes “sea, low horizon” (0.0020). Set \(w = 0\): the condition is ignored, and this noise happens to land on “sea, low horizon” too. Try \(K = 5\): 5 loop steps (10 denoiser passes with guidance) still reach a clean sample here, because the perfect denoiser makes no errors for the solver to collect.

The first backbone: a U-Net with cross-attention

The original LDM and all Stable Diffusion 1.x/2.x/XL models use a U-Net as \(\varepsilon_\theta\). A U-Net is a convolutional encoder–decoder. The down path halves the resolution several times (for a 64×64 latent: 64, 32, 16, 8) while adding channels. The up path doubles it back. Skip connections copy features from each down level to the matching up level, so fine details do not have to pass through the small bottleneck. Each level has residual conv blocks; the noise step \(t\) enters every block as an added embedding. At the lower resolutions there are also attention layers.

The condition enters through cross-attention (session 20): the queries come from the image features, and the keys and values come from the text vectors \(\tau_\phi(y)\):

\[ \text{CrossAttn}(\varphi, \tau) = \softmax\!\left( \frac{(\varphi W_Q)(\tau W_K)^\top}{\sqrt{d}} \right) (\tau W_V) \tag{228.7} \]
Read the symbols
\(\varphi\)
the U-Net features at one level, flattened into a list of \(N\) spatial positions, shape \(N \times d_\varphi\)
\(\tau = \tau_\phi(y)\)
the text vectors, shape \(M \times d_\tau\) (77 tokens × 768 for CLIP ViT-L/14 in SD v1)
\(W_Q, W_K, W_V\)
learned projections into a shared width \(d\)

Each image position asks “which words are about me?” and pulls in their meaning. Stable Diffusion v1 has an 860M-parameter U-Net and a 123M-parameter CLIP ViT-L/14 text encoder, and works at \(f = 8\). [Certain] SDXL made the U-Net about three times larger, mostly through more attention blocks and a wider cross-attention context from a second text encoder. [Certain]

The U-Net has strong inductive biases: built-in assumptions about images. Convolutions are local and treat every position the same way, and the multi-scale design matches the fact that images have structure at many sizes. These biases help with little data. But they also make the network a hand-designed object: how many levels, where to put attention, how many channels at each level. Scaling it up is a craft.

The second backbone: the Diffusion Transformer (DiT)

Peebles & Xie asked a simple question: does the denoiser need the U-Net’s image-specific design at all? They took a standard Vision Transformer (session 164), changed as little as possible, and trained it on latents. The answer was no: with enough compute, the plain transformer is better, and its quality follows a clean scaling rule. [Certain]

Patchify: from latent to tokens

The input is the noisy latent, \(I \times I \times C\) with \(I = 32\) and \(C = 4\) for 256×256 images. DiT cuts it into non-overlapping \(p \times p\) patches, flattens each patch into a vector of \(p^2 C\) numbers, and maps it to width \(d\) with one linear layer. Fixed sine-cosine position embeddings are added. The number of tokens is

\[ T = \left( \frac{I}{p} \right)^{2}, \qquad \text{patch vector length} = p^2 C \tag{228.8} \]

DiT tries \(p \in \{2, 4, 8\}\), giving \(T = 256, 64, 16\). Halving \(p\) multiplies \(T\) by 4 and the compute by at least 4, but leaves the parameter count almost unchanged: only the tiny input and output layers depend on \(p\). [Certain] So the patch size is a pure “compute knob”. Explore the order of the cut in Figure 4.

Patch size \(p\)
Figure 4. Patchify on a toy 8×8 latent with \(C = 4\) channels (the real DiT latent is 32×32×4). Each patch becomes one token. Click any cell to select its patch: the readout shows the token index and the first values of its \(p^2 C\)-long vector, in the order DiT flattens them (row inside the patch, column inside the patch, then channel).
How to read Figure 4

Tokens are numbered row by row over the patch grid, like reading a page. With \(p = 1\), every latent position is a token of only 4 numbers: 64 tokens. With \(p = 8\), the whole latent is one token of 256 numbers, and attention has nothing to attend to. Unpatchify, at the output, is the exact inverse: the same reshape, run backwards.

The DiT block and how the condition gets in

A DiT block is a standard pre-norm transformer block: LayerNorm, self-attention, residual add, LayerNorm, MLP (width \(4d\), GELU), residual add (session 22). The question is how to feed in the step \(t\) and the class \(y\). First both become one vector per image:

\[ \mathbf{c} = \text{MLP}\big(\text{sinusoid}(t)\big) + E_{\text{class}}[y] \in \R^{d} \tag{228.9} \]

DiT compared four ways of using \(\mathbf{c}\), all on DiT-XL/2 [Certain]:

  1. In-context: append \(\mathbf{c}\) as two extra tokens, like a [CLS] token. Almost free.
  2. Cross-attention: add a cross-attention layer to each block that attends to \(\mathbf{c}\) (as in the U-Net with text). Costs about 15% more compute.
  3. adaLN: replace the fixed LayerNorm gain and bias by values computed from \(\mathbf{c}\).
  4. adaLN-Zero: adaLN plus a learned gate on each residual branch, initialized to zero.

Adaptive LayerNorm (adaLN) is the same idea as FiLM: normalize the token, then scale and shift it with numbers that depend on the condition. A small network (SiLU then one linear layer) maps \(\mathbf{c}\) to a shift \(\beta\), a scale \(\gamma\), and a gate \(\alpha\), each of size \(d\), separately for the attention branch and the MLP branch (6 vectors per block):

\[ \text{adaLN}(\mathbf{h}; \mathbf{c}) = \text{LN}(\mathbf{h}) \odot \big(1 + \gamma(\mathbf{c})\big) + \beta(\mathbf{c}) \tag{228.10} \]
\[ \begin{aligned} \mathbf{h} &\leftarrow \mathbf{h} + \alpha_1(\mathbf{c}) \odot \text{Attn}\big(\text{adaLN}_1(\mathbf{h}; \mathbf{c})\big) \\ \mathbf{h} &\leftarrow \mathbf{h} + \alpha_2(\mathbf{c}) \odot \text{MLP}\big(\text{adaLN}_2(\mathbf{h}; \mathbf{c})\big) \end{aligned} \tag{228.11} \]
Read the symbols
\(\text{LN}\)
LayerNorm without its own learned gain and bias: subtract the mean, divide by the standard deviation
\(\gamma, \beta\)
per-channel scale and shift, computed from \(\mathbf{c}\); the same for all \(T\) tokens of one image
\(\alpha\)
per-channel gate on the residual branch; its producing layer starts at zero
\(\odot\)
element-wise multiplication

Why the zero? With \(\alpha = 0\) at the start, every block outputs its input unchanged: the whole network is the identity function. This trick is well known from ResNets (start each residual branch at zero) and makes large models train stably. DiT also zero-initializes the final output layer, so the first prediction is exactly zero. [Certain] Why adaLN instead of cross-attention? The condition is one vector per image, so attending to it adds little; scaling the normalized features is a cheaper and more direct way to say “we are at a high noise level, act accordingly”.

Block design (DiT-XL/2, 400K steps)GflopsFID-50K, no guidance
In-context119.435.24
Cross-attention137.626.14
adaLN118.625.21
adaLN-Zero118.619.47

Numbers from Peebles & Xie, Table 4. [Certain] adaLN-Zero is both the cheapest and the best. Figure 5 lets you move the scale, shift and gate on one toy token.

Worked example: adaLN on one token

Take a token \(\mathbf{h} = (1, 2, 3, 4)\). LayerNorm: mean 2.5, variance 1.25, so \(\text{LN}(\mathbf{h}) = (-1.342, -0.447, 0.447, 1.342)\). Suppose the condition gives \(\gamma = 0.5\) and \(\beta = 0.1\) in every channel. Then formula (228.10) gives \(1.5 \cdot \text{LN}(\mathbf{h}) + 0.1 = (-1.912, -0.571, 0.771, 2.112)\). At a different step \(t\), \(\mathbf{c}\) changes, so \(\gamma\) and \(\beta\) change, and the same weights process the token differently. If the gate \(\alpha = 0\), the block output is \(\mathbf{h} = (1, 2, 3, 4)\) again, whatever the attention or MLP computes.

input token \(\mathbf{h}\)\(\text{adaLN}(\mathbf{h}) = \text{LN}(\mathbf{h})(1 + \gamma) + \beta\)branch \(F(\text{adaLN}(\mathbf{h}))\)output \(\mathbf{h} + \alpha \cdot F\)
Figure 5. One adaLN-Zero residual branch on an 8-dimensional toy token. \(F\) is a fixed random layer (\(\tanh\) of a matrix product) that stands in for attention or the MLP. In a real DiT, \(\gamma\), \(\beta\), \(\alpha\) are vectors computed from \(t\) and \(y\); here each slider sets all 8 channels at once.
How to read Figure 5

With the gate at 0 (its value at initialization), the orange output bars sit exactly on the grey input bars: the block is the identity. Raise the gate and the branch starts to change the token. Move \(\gamma\) to \(-1\): the normalized token is multiplied by 0, so only \(\beta\) passes into the branch, which is how a condition can “switch off” the content of a token at some noise levels.

One whole DiT block, step by step

Figure 6 puts patchify, the condition vector and adaLN-Zero together in one DiT block, run on a noisy toy latent: \(\mathbf{z}_t\) of the “sun left” image from Figure 2, with fixed noise. The block is tiny (\(p = 2\), so \(T = 4\) tokens of \(p^2 C = 16\) numbers, width \(d = 8\), one attention head), and its weights are random numbers from a seeded generator, standing in for trained ones. So the output \(\hat\varepsilon\) is not a good noise prediction; the point is to watch the data flow and the conditioning. Two weight settings: “trained-like” uses random adaLN weights, and “adaLN-Zero init” sets the adaLN layer and the output layer to zero, as DiT does at the start of training.

Class \(y\)Weights
  1. tokens \(\leftarrow\) patchify\((\mathbf{z}_t, p = 2)\) // (228.8): \(4 \times 16\)
  2. \(\mathbf{x} \leftarrow \text{tokens}\, W_e + \text{pos}\) // \(4 \times 8\)
  3. \(\mathbf{c} \leftarrow \text{MLP}(\text{sinusoid}(t)) + E[y]\) // (228.9)
  4. \((\beta_1, \gamma_1, \alpha_1, \beta_2, \gamma_2, \alpha_2) \leftarrow W_{\text{ada}}\, \text{SiLU}(\mathbf{c})\)
  5. \(\mathbf{h} \leftarrow \text{LN}(\mathbf{x}) \odot (1 + \gamma_1) + \beta_1\) // (228.10)
  6. \(\mathbf{x} \leftarrow \mathbf{x} + \alpha_1 \odot \text{Attn}(\mathbf{h})\) // (228.11)
  7. \(\mathbf{x} \leftarrow \mathbf{x} + \alpha_2 \odot \text{MLP}(\text{adaLN}_2(\mathbf{x}))\) // (228.11)
  8. \(\hat\varepsilon \leftarrow\) unpatchify\((\text{Linear}(\text{adaLN}_{\text{out}}(\mathbf{x})))\) // \(4 \times 4 \times 4\)
Figure 6. One DiT block with adaLN-Zero conditioning, computed in your browser. Panels: the noisy latent \(\mathbf{z}_t\) cut into four 2×2 patches; the 4 tokens at the current line (rows are tokens, columns the 8 channels; blue positive, orange negative); the condition \(\mathbf{c}\) and the vectors \(\gamma_1\), \(\beta_1\), \(\alpha_1\) made from it; the attention matrix; and the output \(\hat\varepsilon\) after unpatchify. The live box shows each line with the numbers of token 0, channel 0. “Sweep t” moves \(t\) from 999 to 0 so you can watch the conditioning change.
How to read Figure 6

Step to line 4 and press “Sweep t”: \(\mathbf{c}\), and with it \(\gamma_1\), \(\beta_1\) and \(\alpha_1\), change smoothly with the noise step. The same weights then process the tokens differently at each \(t\). At \(t = 500\) with class sun, the readout shows that the block changes the tokens by \(\lVert \Delta \mathbf{x} \rVert = 1.098\), and the largest output value is 3.403. Change the class to sea: \(\mathbf{c}\) changes (\(c[0]\) goes from −1.416 to −0.375) and the change becomes 0.729. Now choose “adaLN-Zero init”: lines 4 to 8 give \(\gamma = \beta = \alpha = 0\), the attention is still computed (look at the matrix), but nothing is added, so \(\lVert \Delta \mathbf{x} \rVert = 0\) and \(\hat\varepsilon = 0\) exactly, for every \(t\) and class. This is the identity start from the text.

The output layer

After the last block, a final adaLN and a linear layer map each token back to \(p \times p \times 2C\) numbers, and unpatchify puts them back into the latent grid. Why \(2C\) and not \(C\)? DiT predicts both the noise \(\varepsilon\) (\(C\) channels) and the variance of each reverse step (another \(C\) channels), as in Improved DDPM (Nichol & Dhariwal). The noise part is trained with the simple loss (228.4); the variance part with the full variational bound. [Certain] The Hugging Face config of DiT-XL-2-256 shows exactly this: in_channels: 4, out_channels: 8, patch_size: 2, num_layers: 28, 16 heads of size 72 (so \(d = 1152\)), and norm_type: ada_norm_zero. [Certain]

Counting the cost of a DiT

A DiT is a ViT, so we can count its cost with the same rules as an LLM (session 25). Per block, the attention projections have \(4d^2\) weights and the MLP \(8d^2\); every token uses each weight once. The attention scores and the weighted sum add \(2T^2 d\). DiT reports compute in “Gflops” that match multiply-add counts, so per forward pass:

\[ \text{Gflops} \approx L \left( 12\, T d^2 + 2\, T^2 d \right) \cdot 10^{-9} \tag{228.12} \]
\[ \text{params} \approx L \cdot (12 d^2 + 6 d^2) = 18\, L\, d^2 \tag{228.13} \]

The extra \(6d^2\) per block is the adaLN layer that maps \(\mathbf{c}\) to six vectors. It runs once per image, not per token, so it adds parameters but almost no compute.

Worked example: DiT-XL/2

DiT-XL has \(L = 28\), \(d = 1152\); with \(p = 2\), \(T = 256\). Per block: \(12 \cdot 256 \cdot 1152^2 = 4.077 \times 10^9\) for the weights and \(2 \cdot 256^2 \cdot 1152 = 0.151 \times 10^9\) for attention. Times 28 blocks: 118.4 Gflops. The paper reports 118.6 (the difference is the patch embedding, output layer and small terms). Parameters: \(18 \cdot 28 \cdot 1152^2 = 668.9\)M in the blocks; with the embeddings and final layer our count is 675.0M, and the paper says 675M (not counting the 84M VAE). [Certain]

At 512×512 the latent is 64×64, so \(T = 1024\) and the same model needs 524.6 Gflops (formula 228.12 gives 524.3). The pixel-space U-Net ADM needs 1120 Gflops at 256×256 and 1983 at 512×512. [Certain] Sampling with 250 DDPM steps costs \(250 \times 118.6 \approx 29.7\) Tflops per 256×256 image, and twice that with classifier-free guidance.

Scaling behavior of DiT

DiT trained 12 models: four sizes (S, B, L, XL) times three patch sizes (8, 4, 2), all for 400K steps at batch 256 with the same recipe (AdamW, constant learning rate \(10^{-4}\), no weight decay, EMA 0.9999). [Certain]

ModelLayers \(L\)Width \(d\)HeadsParams (our count, 228.13 + extras)Gflops at \(p = 4\)
DiT-S12384633M1.4
DiT-B1276812130M5.6
DiT-L24102416458M19.7
DiT-XL28115216675M29.1

Three findings matter [Certain]:

  1. Compute predicts quality. FID falls steadily as Gflops grow, and it does not matter much whether the Gflops come from a bigger model or from smaller patches (more tokens). The paper calls it a strong negative correlation between Gflops and FID.
  2. Parameters alone do not. DiT-XL/8 has about 675M parameters but only 7.39 Gflops, and its FID is 106.41. DiT-S/2 has about 20 times fewer parameters (33M) and 6.06 Gflops, and its FID is 68.40. Smaller patches, which give more tokens per image, matter more than parameters here.
  3. Bigger models use training compute better. At the same total training compute, larger DiTs reach lower FID. And extra sampling steps cannot make up for a small model: “scaling-up sampling compute cannot compensate for a lack of model compute.”
DiT-SDiT-BDiT-LDiT-XL
Figure 7. The 12 DiT models of Peebles & Xie (Table 4): FID-50K without guidance after 400K steps, against forward Gflops, both on log scales. Lines connect the same model size across patch sizes. The dashed line is our least-squares fit of \(\log \text{FID}\) on \(\log \text{Gflops}\). Click a point or use the buttons.
How to read Figure 7

Points line up along one falling line regardless of color: that is “compute predicts FID”. Compare points that sit at similar Gflops but come from different models, such as B/2 (23.0 Gflops, FID 43.47), L/4 (19.7, 45.64) and XL/4 (29.1, 43.01): very different shapes, similar quality. The readout also checks formula (228.12) against the paper for the selected model.

Reading the fit

A least-squares line through the 12 points in log-log space has slope \(-0.365\) and correlation \(-0.94\). So, in this range, every 10× more forward compute divides FID by about \(10^{0.365} \approx 2.3\). This is our fit on the paper’s 12 numbers, not a law from the paper, and it holds only for this recipe and training length.

With long training (7M steps) and classifier-free guidance at scale 1.5, DiT-XL/2 reached FID 2.27 on ImageNet 256×256, better than LDM-4-G (3.60) and pixel-space ADM. At 512×512 it reached 3.04 after 3M steps. [Certain] Guidance trades diversity for quality, so the guided model has higher precision (0.83) and lower recall (0.57) than the unguided one (0.67 and 0.67). [Certain] More on this trade-off in session 227.

After DiT: text, video and joint models

DiT was class-conditional: the condition was one label. Text needs more than one vector, so later models put cross-attention back in. PixArt-\(\alpha\) adds a cross-attention layer to each DiT block for T5 text features and reports training at about 10.8% of the time of Stable Diffusion v1.5 (675 vs 6,250 A100 GPU-days). [Certain] Stable Diffusion 3 and FLUX go further: text tokens and image tokens are processed in one joint attention over both (MM-DiT), still with adaLN for the timestep, and they train with rectified flow instead of DDPM (session 229). [Likely] U-ViT (Bao et al., 2022) independently showed that a ViT with long skip connections works as a diffusion backbone. [Certain]

For video, the same recipe extends naturally. A video autoencoder compresses both space and time, and the latent is cut into spacetime patches (for example \(2 \times 2\) in space and a few frames in time), giving one long token sequence. OpenAI described Sora this way: a diffusion transformer on spacetime patches of a compressed video latent. [Likely] Because tokens are just a sequence, the same model can train on clips of different lengths, sizes and aspect ratios.

A minimal DiT in PyTorch

The code below builds a small DiT with patchify, adaLN-Zero blocks, the zero-initialized output, and trains it on toy “latents” (one fixed 4×8×8 pattern per class plus noise) with the DDPM loss. Then it samples with DDIM and classifier-free guidance, and checks that each sample is closest to the pattern of its class. In a real model, z0 would be s * vae.encode(x) and the samples would go through vae.decode(z / s). It runs on a CPU in about 15 seconds.

import math, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

def patchify(z, p):                        # (B, C, H, W) -> (B, T, p*p*C), formula (228.8)
    B, C, H, W = z.shape
    z = z.reshape(B, C, H // p, p, W // p, p).permute(0, 2, 4, 3, 5, 1)
    return z.reshape(B, (H // p) * (W // p), p * p * C)

def unpatchify(x, p, C, H, W):             # (B, T, p*p*C) -> (B, C, H, W)
    x = x.reshape(x.shape[0], H // p, W // p, p, p, C).permute(0, 5, 1, 3, 2, 4)
    return x.reshape(x.shape[0], C, H, W)

def t_embed(t, dim=256):                   # sinusoidal timestep features
    half = dim // 2
    freqs = torch.exp(-math.log(10000) * torch.arange(half) / half)
    a = t[:, None].float() * freqs[None]
    return torch.cat([torch.cos(a), torch.sin(a)], dim=-1)

def modulate(x, shift, scale):             # formula (228.10)
    return x * (1 + scale[:, None]) + shift[:, None]

class DiTBlock(nn.Module):
    def __init__(self, d, heads):
        super().__init__()
        self.n1 = nn.LayerNorm(d, elementwise_affine=False, eps=1e-6)
        self.n2 = nn.LayerNorm(d, elementwise_affine=False, eps=1e-6)
        self.attn = nn.MultiheadAttention(d, heads, batch_first=True)
        self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(approximate="tanh"), nn.Linear(4 * d, d))
        self.ada = nn.Sequential(nn.SiLU(), nn.Linear(d, 6 * d))
        nn.init.zeros_(self.ada[1].weight); nn.init.zeros_(self.ada[1].bias)   # adaLN-Zero

    def forward(self, x, c):                # formula (228.11)
        b1, g1, a1, b2, g2, a2 = self.ada(c).chunk(6, dim=-1)
        h = modulate(self.n1(x), b1, g1)
        x = x + a1[:, None] * self.attn(h, h, h, need_weights=False)[0]
        x = x + a2[:, None] * self.mlp(modulate(self.n2(x), b2, g2))
        return x

class TinyDiT(nn.Module):
    def __init__(self, I=8, C=4, p=2, d=64, depth=4, heads=4, n_cls=10):
        super().__init__()
        self.I, self.C, self.p = I, C, p
        T = (I // p) ** 2
        self.x_emb = nn.Linear(p * p * C, d)
        self.pos = nn.Parameter(0.02 * torch.randn(1, T, d))
        self.t_mlp = nn.Sequential(nn.Linear(256, d), nn.SiLU(), nn.Linear(d, d))
        self.y_emb = nn.Embedding(n_cls + 1, d)              # last id = "no class" (for CFG)
        self.blocks = nn.ModuleList([DiTBlock(d, heads) for _ in range(depth)])
        self.n_out = nn.LayerNorm(d, elementwise_affine=False, eps=1e-6)
        self.ada_out = nn.Sequential(nn.SiLU(), nn.Linear(d, 2 * d))
        self.out = nn.Linear(d, p * p * C)                   # real DiT: p*p*2C (noise + variance)
        for m in (self.ada_out[1], self.out):                # zero init: output starts at 0
            nn.init.zeros_(m.weight); nn.init.zeros_(m.bias)

    def forward(self, z, t, y):
        x = self.x_emb(patchify(z, self.p)) + self.pos       # (B, T, d)
        c = self.t_mlp(t_embed(t)) + self.y_emb(y)           # (B, d), formula (228.9)
        for blk in self.blocks:
            x = blk(x, c)
        b, g = self.ada_out(c).chunk(2, dim=-1)
        x = self.out(modulate(self.n_out(x), b, g))          # (B, T, p*p*C)
        return unpatchify(x, self.p, self.C, self.I, self.I) # predicted noise

# DDPM schedule as in DiT: 1000 steps, linear betas from 1e-4 to 2e-2
betas = torch.linspace(1e-4, 2e-2, 1000)
abar = torch.cumprod(1 - betas, dim=0)

model = TinyDiT()
print("params:", sum(q.numel() for q in model.parameters()))        # 331536
z = torch.randn(2, 4, 8, 8)
print("round trip:", torch.equal(unpatchify(patchify(z, 2), 2, 4, 8, 8), z))  # True
print("output at init:", model(z, torch.tensor([10, 500]), torch.tensor([3, 10])).abs().max().item())  # 0.0

# toy "latents": each of 10 classes is a fixed 4x8x8 pattern plus a little noise
protos = torch.randn(10, 4, 8, 8)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.0)
for step in range(1001):
    y = torch.randint(0, 10, (64,))
    z0 = protos[y] + 0.1 * torch.randn(64, 4, 8, 8)       # real LDM: z0 = s * vae.encode(x)
    y_in = torch.where(torch.rand(64) < 0.1, torch.full_like(y, 10), y)   # drop the label 10% of the time
    t = torch.randint(0, 1000, (64,))
    eps = torch.randn_like(z0)
    a = abar[t].view(-1, 1, 1, 1)
    zt = a.sqrt() * z0 + (1 - a).sqrt() * eps              # forward noising, in latent space
    loss = F.mse_loss(model(zt, t, y_in), eps)             # formula (228.4)
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 250 == 0:
        print(step, round(loss.item(), 4))                 # 0.9967 at step 0, below 0.1 by step 1000

# sampling: deterministic DDIM, 50 steps, classifier-free guidance w = 2
@torch.no_grad()
def sample(y, w=2.0, steps=50):
    ts = torch.linspace(999, 0, steps).long()
    zt = torch.randn(len(y), 4, 8, 8)
    for i, t in enumerate(ts):
        tt = t.repeat(len(y))
        e_c = model(zt, tt, y)
        e_u = model(zt, tt, torch.full_like(y, 10))
        e = e_u + w * (e_c - e_u)
        a = abar[t]
        z0_hat = (zt - (1 - a).sqrt() * e) / a.sqrt()
        a_prev = abar[ts[i + 1]] if i + 1 < steps else torch.tensor(1.0)
        zt = a_prev.sqrt() * z0_hat + (1 - a_prev).sqrt() * e
    return zt                                              # real LDM: x = vae.decode(zt / s)

y = torch.arange(10)
zs = sample(y)
dist = ((zs[:, None] - protos[None]) ** 2).mean(dim=(2, 3, 4))   # (10 samples, 10 patterns)
print("nearest pattern:", dist.argmin(1).tolist())             # [0, 1, 2, ..., 9]
print("error to own pattern:", round(dist.diag().mean().item(), 4))  # about 0.018 (data noise alone: 0.01)

Three checks in the output: the patchify round trip is exact, the untrained model outputs exactly 0 (adaLN-Zero plus the zero output layer), and every one of the 10 samples lands nearest to the pattern of the class it was asked for. The loss falls from 0.9967 at step 0 to below 0.1 by step 1,000 (CPU run, PyTorch 2.x; exact later values can differ slightly on other hardware).

Where this is used

System or taskLatent spaceDenoiserNotes
Stable Diffusion 1.x (text-to-image)\(f = 8\), \(c = 4\) KL autoencoder860M U-Net, cross-attention to CLIP textThe model that made LDM popular; runs on a consumer GPU [Certain]
SDXL\(f = 8\), \(c = 4\), \(s = 0.13025\)About 3× larger U-Net, two text encoders[Certain]
DiT (class-conditional ImageNet)Stable Diffusion VAE, 32×32×4 for 256×256Transformer with adaLN-ZeroFID 2.27 with guidance [Certain]
PixArt-\(\alpha\), SD3, FLUX (text-to-image)\(f = 8\); SD3 and FLUX use 16 channels [Likely]DiT with cross-attention (PixArt) or joint text-image attention (MM-DiT)Next session for the training objective
Text-to-video generation (Sora and similar)Video autoencoder that compresses space and timeDiT on spacetime patchesVariable length and aspect ratio come almost for free with tokens [Likely]
Video editing, inpainting and super-resolutionSame latent; known regions are re-noised and keptU-Net or DiT with extra input channels (mask, low-res frame)LDM already did inpainting and super-resolution this way [Certain]
Synthetic training data for video tasks (action recognition, key-moment detection)Any latent video modelAnyGenerate rare actions or camera views to balance a dataset; check for a domain gap [Speculation]
Unified VLMs that also draw (e.g. Transfusion)VAE latent patches inside the LLM sequenceThe LLM transformer itself, with a diffusion loss on image patches[Certain]; see session 232

Common mistakes

  • Forgetting the latent scale \(s\). Training on raw VAE latents (std far from 1) silently changes every noise level of the schedule. Multiply by \(s\) after encoding and divide by \(s\) before decoding, exactly once each.
  • Mixing VAEs. A denoiser trained on one autoencoder’s latents is useless with another autoencoder (for example the 4-channel SD VAE vs a 16-channel VAE), even if the shapes match.
  • Thinking the latent is a nice Gaussian. With KL weight around \(10^{-6}\), the latent is closer to a plain autoencoder code. Interpolating or sampling it directly does not give images; only the diffusion model can.
  • Choosing \(f\) too large. Details lost by the encoder (small text, faces far away) cannot be recovered by the denoiser. When a generator draws bad small text, check the autoencoder’s reconstruction first.
  • Comparing DiTs by parameter count. Patch size changes compute 4× per halving with almost no change in parameters; compare by Gflops (formula 228.12).
  • Using a normal random init for the adaLN layers. Without the zero init, deep DiTs start far from the identity and train worse; the paper’s FID gap between adaLN and adaLN-Zero is 25.21 vs 19.47.
  • Wrong patch order. Patchify and unpatchify must be exact inverses. A mismatched reshape still trains, but the model must learn to undo the scramble, and position embeddings no longer match the content.
  • Running the VAE in FP16 when it overflows. Some VAEs (the SDXL VAE config sets force_upcast: true) produce NaNs in half precision; decode in FP32 or use a fixed VAE. [Likely]

Interview questions

Why do latent diffusion models diffuse in a latent space instead of pixels?basic
Cost and focus. With \(f = 8\), \(c = 4\) the denoiser works on 48 times fewer values, and it runs at every sampling step, so the saving is paid back many times. And most pixel information is perceptual detail that a pretrained autoencoder (trained with perceptual and adversarial losses) can handle once; the diffusion model can then spend its capacity on semantic structure. LDM showed that \(f = 4\) to 8 gives the best trade-off; \(f = 32\) loses too much detail.
Walk me through the shapes in Stable Diffusion v1 for a 512×512 image.basic
Image \(3 \times 512 \times 512\) → VAE encoder (\(f = 8\)) → latent \(4 \times 64 \times 64\), multiplied by 0.18215. Prompt → CLIP ViT-L/14 text encoder → \(77 \times 768\). The U-Net takes the noisy latent \(4 \times 64 \times 64\), the step and the text (via cross-attention), and outputs predicted noise of the same shape. After sampling, divide by 0.18215 and decode back to \(3 \times 512 \times 512\).
What does the scale factor 0.18215 do, and what happens if you forget it?core
It rescales the latent to roughly unit standard deviation, which the noise schedule assumes. Without it, the latent has a larger spread, so at every step the signal-to-noise ratio is much higher than planned: the “fully noised” \(\mathbf{z}_T\) still contains signal, and the start of sampling from pure \(\mathcal{N}(0, I)\) does not match training. Results are washed out or broken. If you forget to divide by it before decoding, the decoder gets latents about 5.5 times too small.
How does DiT turn a latent into tokens, and what does the patch size control?core
It cuts the \(I \times I \times C\) latent into \(p \times p\) patches, flattens each to \(p^2 C\) numbers and projects to width \(d\), then adds fixed sine-cosine position embeddings. \(T = (I/p)^2\): 256 tokens for \(I = 32\), \(p = 2\). Halving \(p\) gives 4× the tokens and at least 4× the compute while parameters stay almost the same, so \(p\) is a compute knob; smaller patches gave much better FID.
Explain adaLN-Zero. Why is it better than cross-attention for class conditioning in DiT?core
The timestep and class embeddings are summed into one vector \(\mathbf{c}\). A linear layer maps \(\mathbf{c}\) to shift, scale and gate vectors for each block. LayerNorm without its own affine parameters is followed by \(\cdot(1 + \gamma) + \beta\), and each residual branch output is multiplied by the gate \(\alpha\). The layer producing these is initialized to zero, so every block starts as the identity. Cross-attention to a single vector adds about 15% compute and brings little; adaLN-Zero is the cheapest and gave the best FID (19.47 vs 26.14 for cross-attention, DiT-XL/2 at 400K steps).
What is the scaling result of the DiT paper?core
Across 12 models (S, B, L, XL × patch 8, 4, 2), FID is strongly predicted by forward Gflops, whether compute comes from depth and width or from more tokens. Parameters alone are a poor predictor: XL/8 (675M params, 7.4 Gflops) has FID 106, worse than S/2 (33M, 6.1 Gflops) at 68. Larger models are more training-compute-efficient, and more sampling steps cannot make up for a smaller model.
Estimate the forward cost of DiT-XL/2 from its config.advanced
\(L = 28\), \(d = 1152\), \(T = 256\). Per block: weights \(12 T d^2 \approx 4.08\)G multiply-adds, attention \(2T^2 d \approx 0.15\)G. Total \(28 \times 4.23 \approx 118.4\) Gflops vs 118.6 reported. Parameters \(\approx 18 L d^2 = 669\)M plus embeddings \(\approx 675\)M. At 512×512, \(T = 1024\) and the attention term grows 16×, giving about 524 Gflops.
Why does DiT output 2C channels instead of C?advanced
It predicts the noise (C channels) and also a per-element variance for the reverse step (C channels), following Improved DDPM. The noise part is trained with the simple MSE; the variance part with the variational lower bound. Learned variances let DDPM sampling use fewer steps with good quality. Models trained with flow matching or used only with DDIM-style samplers usually drop the variance output.
U-Net vs DiT: what are the trade-offs?core
The U-Net has image priors (locality, multi-scale, skips), which help with little data and small compute, and its cost grows roughly linearly with pixels. DiT has few priors; global attention costs \(O(T^2)\) and it needs more data and compute, but it scales predictably, reuses all transformer infrastructure, and handles text and video tokens in the same sequence. At large scale DiTs win; that is why SD3, FLUX and video models use them.
How would you condition a DiT on a text prompt instead of a class?advanced
A prompt is a sequence of vectors, not one vector. Options: (1) add cross-attention to each block, with image tokens as queries and text tokens as keys and values (PixArt-\(\alpha\)); (2) concatenate text and image tokens and use joint self-attention with separate weights per modality (MM-DiT in SD3 and FLUX). Keep adaLN for the timestep and often a pooled text vector. For classifier-free guidance, drop the text (use an empty prompt) for about 10% of training examples.
Your latent diffusion model draws blurry small text and faces. Where do you look first?advanced
At the autoencoder: encode and decode real images and inspect the reconstructions. If small text is already broken there, no denoiser can fix it; you need a better autoencoder (more channels, e.g. 16, or smaller \(f\)), or a fine-tuned decoder. If reconstructions are fine, look at the denoiser: resolution of training data, guidance scale, number of steps, model size.

Exercises

  1. Compute the latent shape and the compression ratio for a 768×768 image with \(f = 8\), \(c = 4\), and with \(f = 8\), \(c = 16\).
  2. A DiT-B (\(L = 12\), \(d = 768\)) runs on a 32×32×4 latent with \(p = 4\). How many tokens? Estimate the Gflops with formula (228.12) and compare with Figure 7.
  3. Show that if \(\alpha = 0\) in every block and the output layer is zero, the DiT output is 0 for any input. Why does this not stop learning?
  4. You want 256×256 images with a DiT-L but can only afford about 20 Gflops per forward. Which patch size? What FID does Table 4 report at 400K steps?
  5. A latent has standard deviation 5.5 instead of 1. At a step where \(\bar\alpha_t = 0.5\), what is the signal-to-noise ratio \(\bar\alpha_t \sigma_z^2 / (1 - \bar\alpha_t)\), compared to the intended value?
  6. Count the tokens of a video DiT for 16 frames of 256×256 with a video autoencoder that compresses 8× in space and 4× in time, and patches of \(2 \times 2\) in space and 1 in time.
Answerscheck
1. \(96 \times 96 \times 4 = 36{,}864\) values vs \(768 \cdot 768 \cdot 3 = 1{,}769{,}472\): ratio 48. With \(c = 16\): \(96 \times 96 \times 16 = 147{,}456\), ratio 12. 2. \(T = (32/4)^2 = 64\). \(12 \cdot 12 \cdot 64 \cdot 768^2 + 12 \cdot 2 \cdot 64^2 \cdot 768 = 5.44 + 0.08 \approx 5.5\) Gflops; the paper reports 5.56. 3. With \(\alpha = 0\), each block returns \(\mathbf{h}\) unchanged; the zero output layer then maps everything to 0. Gradients still flow: the gradient with respect to the output layer’s weights is not zero (it depends on the features and the loss), and once those weights move, gradients reach the \(\alpha\) layers, which then open the branches. 4. \(p = 4\): DiT-L/4 has 19.70 Gflops and FID 45.64 (L/2 is 80.71 Gflops, L/8 only 5.01 with FID 118.87). 5. Intended: \(0.5 \cdot 1/0.5 = 1\). Actual: \(0.5 \cdot 30.25/0.5 = 30.25\), about 30 times more signal than the schedule assumes. 6. Latent: \(16/4 = 4\) frames of \(32 \times 32\); tokens \(4 \cdot (32/2)^2 = 4 \cdot 256 = 1{,}024\).

Key takeaways

  • Latent diffusion splits generation into perceptual compression (an autoencoder, trained once) and semantic generation (diffusion on the small latent), formula (228.1). Stable Diffusion: \(f = 8\), \(c = 4\), 48× fewer values (228.2).
  • The diffusion math is unchanged; only the data is \(\mathbf{z}_0 = s\,\mathcal{E}(\mathbf{x})\) (228.4). The scale \(s\) (0.18215 for SD 1.x) keeps the noise schedule valid.
  • The U-Net backbone uses convolutions, skips and cross-attention (228.7); DiT replaces it with a plain ViT on latent patches (228.8).
  • DiT conditions through adaLN-Zero (228.10, 228.11): scale, shift and gate computed from the timestep and class; zero init makes every block start as the identity.
  • DiT quality follows forward compute (228.12), not parameter count; smaller patches and bigger models both help, and bigger models use training compute better. DiT-XL/2: 675M params, 118.6 Gflops, FID 2.27 with guidance.

Key formulas

\[ \frac{3f^2}{c}, \qquad \mathcal{L} = \lVert \varepsilon - \varepsilon_\theta(\mathbf{z}_t, t, y) \rVert^2, \qquad T = \Big(\frac{I}{p}\Big)^2, \qquad \text{LN}(\mathbf{h})(1 + \gamma) + \beta, \qquad L(12Td^2 + 2T^2 d) \]

References for this session

  1. Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models
  2. Peebles & Xie (2022). Scalable Diffusion Models with Transformers
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 10: Any-to-Any Models
  4. Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models
  5. Peebles & Xie (2022). Scalable Diffusion Models with Transformers
  6. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 10: Any-to-Any Models
  7. Ho et al. (2020). Denoising Diffusion Probabilistic Models
  8. Song et al. (2020). Denoising Diffusion Implicit Models
  9. Ho & Salimans (2022). Classifier-Free Diffusion Guidance
  10. Dhariwal & Nichol (2021). Diffusion Models Beat GANs on Image Synthesis
  11. Esser et al. (2020). Taming Transformers for High-Resolution Image Synthesis
  12. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  13. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  14. Ronneberger, Fischer & Brox (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation
  15. Nichol & Dhariwal (2021). Improved Denoising Diffusion Probabilistic Models
  16. Perez et al. (2017). FiLM: Visual Reasoning with a General Conditioning Layer
  17. Bao et al. (2022). All are Worth Words: A ViT Backbone for Diffusion Models
  18. Chen et al. (2023). PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis
  19. Podell et al. (2023). SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
  20. Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
  21. Brooks et al. (OpenAI) (2024). Video generation models as world simulators (technical report)
  22. Zhou et al. (2024). Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model