LLM & VLM courseBlog
231Track 9: Generative models · session 10 of 12

Video generation: temporal coherence, video DiTs and evaluation

A video is a stack of images that must agree with each other over time. This session shows how modern video generators compress a clip into a grid of space-time tokens, run a diffusion transformer with attention over all of them, keep objects the same from frame to frame, and how we measure the result (FVD, benchmarks, human preference, physics checks). We also count the cost: a 5-second 720p clip is 75,600 tokens, and attention is most of the compute.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Diffusion 1 and 2, DiT and latent diffusion, flow matching (sessions 225 to 230); attention and RoPE; basic PyTorch

What you will learn

  • How a causal 3D VAE and a 1×2×2 patchify turn a clip into space-time tokens, and how to count them
  • Full 3D attention versus factorized (spatial then temporal) attention: what each can see and what each costs
  • Why frames generated separately flicker, and which tools keep identity consistent: joint attention, 3D RoPE, joint image-video training, first-frame conditioning, reconstruction guidance
  • The anatomy of a video DiT (Wan, CogVideoX, HunyuanVideo): text conditioning, adaLN timestep, flow-matching loss
  • FVD: what it measures, its small-sample bias and its bias toward single-frame quality; VBench, human preference with Elo, physics and motion checks
  • How to estimate FLOPs, memory and latency for a video sample, and which levers cut them (fewer steps, sparse attention, caching, context parallelism)

Why this matters

An image generator makes one picture. A video generator must make many pictures that agree with each other: the same face in every frame, a ball that falls instead of jumping, a camera that moves smoothly. This one extra requirement, temporal coherence, changes the architecture, the cost and the way we evaluate.

In the previous sessions you built every piece of an image generator: the forward noising process and the denoiser (session 225, session 226), text conditioning and classifier-free guidance (session 227), latent diffusion with a transformer backbone, flow matching, and few-step sampling (session 230). Video generation reuses all of them. What is new is the time axis. It multiplies the number of tokens by 20 or more, it creates a new kind of error (flicker and identity drift), and it makes the old image metrics blind to the most visible failures.

The models you will meet in practice share one design. Wan 2.1, CogVideoX, HunyuanVideo, Movie Gen and Sora all compress the clip with a video autoencoder, cut the latent into space-time patches, and run a diffusion transformer (DiT) over the patches of all frames. [Likely] The open models publish their configurations, so we can count their tokens and FLOPs exactly; we do this below for Wan 2.1-14B.

Video matters beyond creative tools. Generated clips are used as training data for video understanding models (action recognition, video retrieval, key-moment detection), as world models for robots and driving simulators, and for video editing. And the same space-time token machinery appears in video encoders (session 202) and video LLMs (session 205). The plan: space-time latents and tokens, attention over space and time, the video DiT, temporal coherence, evaluation, and cost.

From images to video: space-time latents

A raw video is huge. Five seconds at 16 frames per second (fps) and 720 × 1280 pixels is 81 frames × 720 × 1280 × 3 colors = 223,948,800 numbers. No transformer can attend over that directly. The first step is the same as in latent image diffusion: an autoencoder compresses the clip into a much smaller latent tensor, the diffusion model works in that latent space, and a decoder turns the result back into pixels.

For video, the autoencoder compresses time as well as space. A 3D VAE uses 3D convolutions (over frames, height and width). The common choice in 2024 and 2025 models is a factor 4 in time and 8 in each spatial direction, with 16 latent channels: Wan, CogVideoX and HunyuanVideo all publish exactly these numbers in their configs. [Certain]

The VAE is also causal in time: each latent frame only looks at the current and earlier video frames. The first frame is compressed alone (only spatially), and every later group of 4 frames becomes one latent frame. [Certain] This is why these models like frame counts of the form \(1 + 4k\) (49, 81, 121), and it has a useful side effect: a single image is just a video with one frame, so the same VAE and the same DiT can be trained on images and videos together.

After the VAE, the DiT cuts the latent into patches. Wan uses a 3D convolution with kernel and stride \(1 \times 2 \times 2\): one latent frame in time, 2 × 2 latent pixels in space. [Certain] Each patch becomes one token. Putting the two steps together:

\[ N \;=\; \underbrace{\Big(1 + \frac{F - 1}{c_t}\Big)}_{T \text{ latent frames}} \cdot \underbrace{\frac{H}{c_s\, p}}_{h} \cdot \underbrace{\frac{W}{c_s\, p}}_{w} \tag{231.1} \]
Read the symbols
\(F, H, W\)
number of video frames, and frame height and width in pixels
\(c_t, c_s\)
VAE compression in time (4) and in each spatial direction (8)
\(p\)
spatial patch size of the DiT (2); the temporal patch size is 1
\(T, h, w\)
the token grid: \(T\) latent frames, each with \(h \times w\) patch tokens; we write \(S = h w\) for the tokens per latent frame
\(N\)
the sequence length of the transformer: \(N = T \cdot S\)
Worked example: how many tokens is a 5-second 720p clip?

Wan 2.1 defaults: \(F = 81\) frames at 16 fps (5 s), \(H \times W = 720 \times 1280\).

Time: \(T = 1 + 80/4 = 21\) latent frames. Space: \(720/8 = 90\) and \(1280/8 = 160\) latent pixels, so the latent is \(21 \times 90 \times 160 \times 16 = 4{,}838{,}400\) numbers, 46.3 times fewer than the 223,948,800 pixel values. Patches: \(90/2 = 45\), \(160/2 = 80\), so \(S = 3{,}600\) tokens per latent frame and

\(N = 21 \times 3{,}600 = 75{,}600\) tokens.

For comparison, a 1024 × 1024 image with the same VAE and patch size is \(64 \times 64 = 4{,}096\) tokens. The clip is 18.5 times longer. At 480 × 832 the same clip is \(21 \times 30 \times 52 = 32{,}760\) tokens; CogVideoX-5B at 49 frames and 480 × 720 is \(13 \times 30 \times 45 = 17{,}550\) video tokens. Movie Gen reports a maximum context of 73K video tokens for a 16-second clip at 16 fps. [Certain]

Figure 1 shows the whole path from pixels to tokens and back.

81 frames 720 × 1280 × 3 causal 3D VAE encoder ÷4 time, ÷8 space latent 21 × 90 × 160 × 16 1×2×2 … 75,600 tokens + 3D RoPE (t, y, x) DiT block × L 3D self-attention cross-attn (text) MLP text tokens (umT5) time t → adaLN scale/shift ×50 steps unpatchify → 3D VAE decoder → 81 frames video
Figure 1. The standard latent video DiT, with the numbers of Wan 2.1 for a 5-second 720p clip. For text-to-video, the VAE decoder runs once at the end (the encoder is used for training data, editing and image-to-video); the DiT runs once per sampling step, or twice with classifier-free guidance.

Attention over space and time

Inside the DiT, each token must learn from other tokens. The question is: which other tokens? There are two main answers, and the choice sets both the quality and the cost.

  • Factorized (divided) attention. Alternate two cheaper attentions. Spatial attention: each token attends only to the \(S\) tokens of its own frame. Temporal attention: each token attends only to the \(T\) tokens at the same spatial position in all frames. This is the design of Video Diffusion Models (Ho et al., 2022), which inserts a temporal attention block after each spatial attention block of an image U-Net, and of video encoders such as TimeSformer and ViViT. [Certain] Latte studies four ways to split space and time inside a DiT. [Certain]
  • Full 3D attention. Every token attends to every other token of the clip, in one big attention over all \(N = T S\) tokens. CogVideoX, HunyuanVideo, Movie Gen and Wan use it [Likely]; the Wan report explicitly calls it “full spatio-temporal attention”. [Certain]

Count the score entries (query-key pairs) for one attention layer:

\[ \text{full: } N^2 = T^2 S^2, \qquad \text{factorized: } \underbrace{T \cdot S^2}_{\text{spatial}} \;+\; \underbrace{S \cdot T^2}_{\text{temporal}} \tag{231.2} \]

For the 720p clip (\(T = 21\), \(S = 3{,}600\)): full attention has \(5.72 \times 10^9\) entries per head; one spatial plus one temporal layer has \(2.74 \times 10^8\), about 21 times fewer. That is a big saving. So why did the field move to full attention?

Because factorized attention is indirect. A token at frame 0 sees a token at frame 5 in a different place only through a two-hop path: first spatial (to the token at the same place in frame 0), then temporal. Information about a moving object must be passed along these axis-aligned paths. When an object moves fast, the “same position in the other frame” holds background, not the object. Full attention can match the object directly wherever it is. In practice, the large open models that report both designs found full attention gives better motion and consistency, and pay the cost with FlashAttention and parallelism. [Likely] Figure 2 shows what one query can see under each pattern.

Figure 2. A toy clip of 6 latent frames with 4 × 4 tokens each (96 tokens). Each mode shows which keys one query attends to. “3D window ±1” is a local window of one frame and one token in each direction, the idea behind sliding-window and sliding-tile attention.
How to read Figure 2

Click any cell to move the query. Full attention lights every cell (96 keys, 9,216 score entries for the clip). Spatial lights one frame (16 keys); temporal lights one position in each frame (6 keys). Spatial plus temporal together still connect every pair of tokens, but only in two hops. The window mode is cheap (at most 27 keys) and keeps the most useful neighbors; Sliding Tile Attention found that the attention of a pretrained video DiT is mostly concentrated in such local 3D windows. [Certain] Move the query to a corner and the window shrinks to 8 keys.

Telling the model where each token is: 3D RoPE

Attention by itself does not know the order of tokens. Video DiTs use 3D rotary position embedding (RoPE): the per-head query and key vectors are split into three parts, and each part is rotated by an angle that depends on one coordinate (frame \(t\), row \(y\), column \(x\)). For a pair of dimensions \(j\) in the time part:

\[ \begin{pmatrix} q'_{2j} \\ q'_{2j+1} \end{pmatrix} = \begin{pmatrix} \cos t\theta_j & -\sin t\theta_j \\ \sin t\theta_j & \cos t\theta_j \end{pmatrix} \begin{pmatrix} q_{2j} \\ q_{2j+1} \end{pmatrix}, \qquad \theta_j = b^{-2j/d_t} \tag{231.3} \]
Read the symbols
\(d_t, d_y, d_x\)
how many of the head dimensions encode time, rows and columns; Wan splits its 128 dimensions as 44 / 42 / 42, HunyuanVideo as 16 / 56 / 56 [Certain]
\(\theta_j\)
the rotation speed of pair \(j\); \(b\) is the base (10,000 in Wan)
\(q'\)
the rotated query; keys are rotated the same way with their own coordinates

The key property: the dot product of a rotated query and a rotated key depends only on the offset \((\Delta t, \Delta y, \Delta x)\) between them, not on their absolute positions (the PyTorch code below checks this numerically). So “the same object one frame later and two patches to the right” looks the same to the model anywhere in the clip, which helps it track motion, and the model can be run on other lengths and sizes than it saw most in training. [Likely]

The video DiT

A video DiT is the DiT of Peebles and Xie with three changes: tokens come from a 3D latent, positions are 3D, and the text prompt is long. Each block has self-attention over all video tokens, a way to read the text, and an MLP; the diffusion time \(t\) enters through adaLN (adaptive layer norm: the time embedding predicts a scale, a shift and a gate for each sub-layer, the adaLN-Zero idea of DiT). [Certain] There are two common ways to add text:

  • Cross-attention (Wan): video tokens are the queries, and the 512 text tokens of a umT5 encoder are the keys and values. [Certain] The text does not grow the self-attention.
  • Joint attention (CogVideoX, HunyuanVideo, and the MM-DiT of Stable Diffusion 3): text and video tokens are concatenated into one sequence and share the self-attention, with separate weights or norms per modality. CogVideoX calls its version “expert adaptive LayerNorm”; HunyuanVideo uses 20 dual-stream blocks followed by 40 single-stream blocks. [Certain]

Training is the same as for images, now on the whole clip at once. With the flow-matching (rectified flow) objective used by Wan [Certain] and HunyuanVideo [Likely]:

\[ \mathbf{z}_t = (1 - t)\,\mathbf{z}_0 + t\,\boldsymbol{\varepsilon}, \qquad \mathcal{L}(\theta) = \E_{\mathbf{z}_0, \boldsymbol{\varepsilon}, t, c}\, \big\lVert \mathbf{v}_\theta(\mathbf{z}_t, t, c) - (\boldsymbol{\varepsilon} - \mathbf{z}_0) \big\rVert^2 \tag{231.4} \]
Read the symbols
\(\mathbf{z}_0\)
the clean latent of the whole clip, shape \(T \times h' \times w' \times 16\) (one noise level for all frames)
\(\boldsymbol{\varepsilon}\)
Gaussian noise of the same shape, independent per frame and per position
\(t \in [0, 1]\)
the noise level (Wan samples it from a logit-normal distribution [Certain]); \(t = 1\) is pure noise
\(\mathbf{v}_\theta\)
the network’s predicted velocity; the target \(\boldsymbol{\varepsilon} - \mathbf{z}_0\) points from data to noise
\(c\)
the condition: text tokens, and for image-to-video also the first frame

Sampling integrates the learned ODE from noise to data, with classifier-free guidance as in session 227:

\[ \tilde{\mathbf{v}} = \mathbf{v}_\theta(\mathbf{z}_t, t, \varnothing) + w\,\big(\mathbf{v}_\theta(\mathbf{z}_t, t, c) - \mathbf{v}_\theta(\mathbf{z}_t, t, \varnothing)\big), \qquad \mathbf{z}_{t - \Delta} = \mathbf{z}_t - \Delta\, \tilde{\mathbf{v}} \tag{231.5} \]

Here \(\varnothing\) is the empty (or negative) prompt and \(w\) the guidance scale. Each step costs two forward passes over all \(N\) tokens. The Wan code defaults to 50 steps. [Certain]

Algorithm 1: one forward pass of a video DiT
  1. input: noisy latent \(\mathbf{z}_t\) of shape \(16 \times T \times h' \times w'\); time \(t\); text tokens \(c\) (512 × 4096 for umT5)
  2. \(\mathbf{x} \leftarrow \text{Conv3d}_{1\times2\times2}(\mathbf{z}_t)\), flatten to \(N \times D\) // formula (231.1): \(N = T \cdot h \cdot w\) tokens
  3. \(\mathbf{e} \leftarrow \text{MLP}(\text{sinusoid}(t))\); one shared MLP gives scale, shift, gate for every block // adaLN
  4. for each of the \(L\) blocks:
  5.   \(\mathbf{q}, \mathbf{k}, \mathbf{v} \leftarrow\) projections of \(\text{adaLN}(\mathbf{x}, \mathbf{e})\); rotate \(\mathbf{q}, \mathbf{k}\) by 3D RoPE // (231.3)
  6.   \(\mathbf{x} \leftarrow \mathbf{x} + \text{gate} \cdot \text{FlashAttention}(\mathbf{q}, \mathbf{k}, \mathbf{v})\) over all \(N\) tokens // full 3D attention
  7.   \(\mathbf{x} \leftarrow \mathbf{x} + \text{CrossAttention}(\text{norm}(\mathbf{x}),\ c)\) // read the prompt
  8.   \(\mathbf{x} \leftarrow \mathbf{x} + \text{gate} \cdot \text{MLP}(\text{adaLN}(\mathbf{x}, \mathbf{e}))\)
  9. return \(\text{unpatchify}(\text{Linear}(\text{adaLN}(\mathbf{x}, \mathbf{e})))\) // velocity, same shape as \(\mathbf{z}_t\)
Algorithm 2: text-to-video sampling
  1. input: prompt, frame count \(F = 1 + 4k\), size \(H \times W\), steps \(K\), guidance \(w\)
  2. \(c \leftarrow \text{TextEncoder}(\text{prompt})\); \(\ c_\varnothing \leftarrow \text{TextEncoder}(\text{negative prompt})\)
  3. \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\) of shape \(16 \times (1 + (F-1)/4) \times H/8 \times W/8\) // noise for every frame at once
  4. for \(k = 1, \dots, K\) with times \(1 = t_1 \gt \dots \gt t_{K+1} = 0\): // a shifted schedule spends more steps at high noise
  5.   \(\tilde{\mathbf{v}} \leftarrow\) guided velocity (231.5) from two DiT passes (Algorithm 1)
  6.   \(\mathbf{z} \leftarrow \mathbf{z} - (t_k - t_{k+1})\, \tilde{\mathbf{v}}\) // Euler step; higher-order solvers also work
  7. return \(\text{VAEDecode}(\mathbf{z})\), in temporal chunks with a feature cache to bound memory // causal VAE makes chunking exact

Keeping objects consistent across frames

Why not simply run a good image model 81 times? Because the image model samples each frame independently. Each frame can look perfect, yet the person’s shirt changes color, the dog turns into another dog, and textures “boil” from frame to frame. Even with the same starting noise, small differences between frames push the sampler to different decisions. This failure is called flicker or identity drift.

The cause is precise. A diffusion sampler makes its big decisions (which object, which layout) at high noise, and its small decisions (texture) at low noise. If frame \(f\)’s denoiser cannot see the other frames when it makes the big decision, the frames will disagree. The fix is to let all frames make that decision together. Figure 3 shows this in a toy model where everything can be computed exactly.

The toy: a “video” is 16 frames, each frame one number (the position of an object). The clip shows either object A (position near +1) or object B (near −1), with probability 1/2 each, and the same object in all frames: \(x_f = m + 0.25\,e_f\), \(m \in \{-1, +1\}\), \(e_f \sim \mathcal{N}(0, 1)\). Real video has this structure: the content is shared across frames, and only small details change. We sample with the exact denoiser of a model whose frame \(f\) can see only the \(2w + 1\) frames around it (its temporal context):

\[ D_f(\mathbf{x}; \sigma) = \hat m_f \Big(1 - \frac{s^2}{v}\Big) + \frac{s^2}{v}\, x_f, \qquad \hat m_f = \tanh\!\Big(\frac{1}{v} \sum_{|j - f| \le w} x_j\Big), \quad v = s^2 + \sigma^2 \tag{231.6} \]
Read the symbols
\(\hat m_f\)
the posterior mean of the object identity \(m\) given the noisy frames in the window; \(\tanh\) comes from Bayes’ rule with two equally likely objects
\(s = 0.25\)
the per-frame variation around the object’s position
\(w\)
the temporal context: \(w = 0\) is an image model run frame by frame; \(w = 15\) sees the whole clip, like full 3D attention
Figure 3. 300 clips sampled with the exact denoiser (231.6) and a 30-step Euler ODE sampler from \(\sigma = 20\) (Karras schedule, \(\rho = 7\)), all from the same seeded starting noise. Left: the first 12 clips; each cell is one frame, gray for object A and orange for object B. Right: average number of identity switches between neighboring frames, for every context size.
How to read Figure 3

At \(w = 0\) every frame chooses its object alone: about 7.5 switches per clip (15 neighbor pairs, each a coin flip), and no clip keeps one object. Yet the per-frame quality in the readout stays between 94% and 97% for every \(w\): each frame alone looks real. Raise \(w\): switches drop fast, but clips still break into blocks, because frames more than \(2w\) apart never share a decision. Only when the context covers the clip (\(w = 15\)) are all clips consistent. The lesson for real models: consistency needs every frame to see the others at the noise levels where content is decided, which is what full 3D attention gives. It also warns that any metric computed frame by frame cannot see this failure.

Real systems combine several tools to keep identity stable:

  • Joint denoising of the whole clip. All frames share one sampling trajectory and attend to each other at every step. This is the main tool, and the main reason for full attention. [Likely]
  • Joint image and video training. Ho et al. found that training on images and videos together improves sample quality; the image batches teach appearance, the videos teach motion. [Certain] With a causal VAE, an image is simply a one-frame video.
  • Start from an image model, then add time. Align Your Latents and Stable Video Diffusion turn a pretrained image latent diffusion model into a video model by inserting temporal layers and fine-tuning on video. [Certain] The spatial knowledge carries over; only motion must be learned.
  • Condition on a frame (image-to-video). Give the model the first frame as an extra input (for example, its latent concatenated to the noisy latent, with a mask). The identity is then fixed by the condition, not sampled. [Likely]
  • A video VAE, not an image VAE. Decoding each frame with an image VAE adds its own small, independent errors per frame, which shows up as flicker in fine texture. The 3D VAE decodes frames together. [Likely]
  • Generate the whole clip at once rather than keyframes plus interpolation. Lumiere argues that making distant keyframes first and filling in between makes global consistency hard, and generates the full frame rate in one pass. [Certain]

Longer than the training window

Models are trained on clips of a fixed length (5 to 16 seconds). For longer videos, the usual answer is autoregressive extension: generate a clip, then generate the next clip conditioned on the last frames of the previous one. The naive version, called the replacement method, fixes the known frames to their (noised) true values at each step and samples only the new frames. Ho et al. found that this gives new frames that look good alone but are not coherent with the old ones, because the denoiser of the new frames does not account for the old frames properly. [Certain] Their fix, reconstruction guidance, adds a gradient that pulls the model’s reconstruction of the known frames toward the true ones:

\[ \tilde{\mathbf{x}}^b_\theta(\mathbf{z}_t) = \hat{\mathbf{x}}^b_\theta(\mathbf{z}_t) - \frac{w_r \alpha_t}{2}\, \nabla_{\mathbf{z}^b_t} \big\lVert \mathbf{x}^a - \hat{\mathbf{x}}^a_\theta(\mathbf{z}_t) \big\rVert_2^2 \tag{231.7} \]
Read the symbols
\(\mathbf{x}^a\)
the known frames (for example the end of the previous clip)
\(\mathbf{x}^b\)
the new frames being generated; \(\mathbf{z}^b_t\) is their noisy version
\(\hat{\mathbf{x}}^a_\theta, \hat{\mathbf{x}}^b_\theta\)
the model’s denoised estimates of both parts, from one pass over the joint clip
\(w_r\)
the guidance weight; values above 1 tend to improve quality [Certain]; \(\alpha_t\) is the signal scale of the noise schedule

Autoregressive extension has a known problem: small errors accumulate from chunk to chunk, so colors drift and objects slowly change. Diffusion Forcing trains with an independent noise level per frame, so a causal model can roll out past its training length while keeping past frames only partly noisy; CausVid converts a bidirectional video DiT into a causal one and distills 50 steps into 4 for streaming generation. [Certain] These ideas connect video generation back to next-token prediction.

Evaluation: FVD, benchmarks, people and physics

Evaluating a video model needs four kinds of question. Does each frame look real? Does the motion look real? Does the clip match the prompt? Does it obey physics? No single number answers all four.

Fréchet Video Distance

FVD (Unterthiner et al., 2018) is FID for video. Pass real and generated clips through a pretrained video classifier, the I3D network trained on Kinetics-400; the paper found its logits layer works best. [Certain] Fit a Gaussian to each set of feature vectors and compute the Fréchet distance between the two Gaussians:

\[ \text{FVD} = \lVert \boldsymbol{\mu}_r - \boldsymbol{\mu}_g \rVert^2 + \operatorname{tr}\!\Big( \Sigma_r + \Sigma_g - 2\big(\Sigma_r \Sigma_g\big)^{1/2} \Big) \tag{231.8} \]
Read the symbols
\(\boldsymbol{\mu}_r, \Sigma_r\)
mean and covariance of the I3D features of real clips
\(\boldsymbol{\mu}_g, \Sigma_g\)
the same for generated clips
\((\Sigma_r \Sigma_g)^{1/2}\)
a matrix square root; for one-dimensional features the whole formula becomes \((\mu_r - \mu_g)^2 + (\sigma_r - \sigma_g)^2\)

Lower is better, and 0 means the two Gaussians are equal. In a human study, FVD agreed with human raters better than PSNR, SSIM and frame-averaged FID. [Certain] But it has two well-known problems.

Problem 1: it depends on the sample size. The means and covariances are estimated from \(n\) clips, so even two sets from the same distribution give FVD \(\gt 0\). The FVD paper says to always compare at the same sample size. [Certain] For unit-variance Gaussian features in \(d\) dimensions, the bias is about

\[ \E\big[\widehat{\text{FD}}\big] \;\approx\; \text{FD} + \frac{2d}{n} + \frac{d(d+1)}{2n} \;=\; \text{FD} + \frac{d(d+5)}{2n} \tag{231.9} \]

The first term comes from the noise in the two means (exactly \(2d/n\) in expectation), the second from the covariances. We checked the formula numerically: with \(d = 16\) and \(n = 256\), the measured mean over 20 trials is 0.674 against a prediction of 0.656; at \(d = 32\), \(n = 256\), 2.299 against 2.312. The bias grows with the square of the dimension and falls as \(1/n\). Figure 4 lets you try it.

estimated FD from \(n\) samplestrue \(\text{FD} = \delta^2\)\(\delta^2 + d(d+5)/(2n)\)
Figure 4. Real features are \(\mathcal{N}(\mathbf{0}, I_d)\); generated features are the same Gaussian shifted by \(\delta\) along the first axis, so the true Fréchet distance is \(\delta^2\). Each point draws \(n\) real and \(n\) generated vectors with a fixed seed and computes (231.8) exactly (matrix square root by eigen-decomposition).
How to read Figure 4

With \(\delta = 0\) the two distributions are identical, yet the estimate is far above 0 at small \(n\) and follows the dashed prediction down as \(n\) grows. Set \(\delta = 0.5\) (true FD 0.25) and \(d = 32\): at \(n = 64\) the bias (about 9.3) hides the real difference completely. Now imagine 400-dimensional I3D logits: comparing a model scored on 256 clips with one scored on 2,048 clips is meaningless. Always report \(n\) and use the same \(n\) for every model.

Problem 2: content bias. Ge et al. (2024) showed that FVD reacts strongly to the quality of single frames and only weakly to broken motion: large temporal corruption raised FVD only a little, and picking generated videos with no motion from a large set lowered FVD without improving temporal quality. They trace this to I3D, a supervised classifier trained on a dataset where appearance often reveals the action, and find that features from large self-supervised video models are less biased. [Certain] Figure 3 makes the same point in miniature: per-frame quality is identical with and without flicker.

Worked example: Fréchet distance in one dimension

Real features \(\mathcal{N}(0, 1)\); generated features \(\mathcal{N}(0.5, 2^2)\). Then (231.8) in one dimension is \((\mu_r - \mu_g)^2 + \sigma_r^2 + \sigma_g^2 - 2\sigma_r\sigma_g = 0.25 + 1 + 4 - 4 = 1.25\). The mean shift contributes 0.25 and the wrong spread contributes \((1 - 2)^2 = 1\). A model that generates too-diverse (or too-uniform) clips is penalized even if its average is right.

Benchmarks with many dimensions

Because one number hides too much, modern evaluation splits quality into many measured dimensions. VBench has 16, including subject consistency, background consistency, temporal flickering, motion smoothness, dynamic degree, aesthetic and imaging quality, object class, spatial relationship and overall consistency with the prompt. [Certain] Each uses a specialist model. For example, subject consistency compares DINO features of each frame with the first frame and the previous frame:

\[ S_{\text{subject}} = \frac{1}{T - 1} \sum_{t=2}^{T} \frac12 \Big( \langle \mathbf{d}_1, \mathbf{d}_t \rangle + \langle \mathbf{d}_{t-1}, \mathbf{d}_t \rangle \Big) \tag{231.10} \]

where \(\mathbf{d}_t\) is the normalized DINO feature of frame \(t\). [Certain] Temporal flickering is the mean absolute pixel difference between consecutive frames of static clips; motion smoothness drops every other frame, rebuilds it with a frame-interpolation model and measures the error; dynamic degree uses RAFT optical flow to decide if a clip moves at all. [Certain]

Notice the tension: a clip that does not move scores perfectly on consistency, flicker and smoothness. That is why dynamic degree exists, and why you must read consistency and motion together. The Wan report uses its own Wan-Bench with 14 fine-grained metrics, including large motion, physical plausibility and ID consistency. [Certain]

Human preference

The final judge is people. A common protocol shows raters two clips for the same prompt and asks which is better (overall, or per aspect: prompt alignment, visual quality, motion). Many pairwise votes are turned into a single score with the Bradley–Terry model, the same idea as Elo ratings in chess and in model arenas:

\[ P(A \text{ beats } B) = \frac{1}{1 + 10^{(R_B - R_A)/400}} \tag{231.11} \]

A model rated 100 points higher wins 64.0% of comparisons; 200 points higher wins 76.0%. Human studies are slow and noisy, so reports should give the number of prompts and raters and the confidence interval. The Wan report, for example, used over 700 evaluation tasks rated by more than 20 people. [Certain]

Physics and motion checks

A clip can be sharp, smooth and on-prompt and still be physically wrong: a glass that does not break, water that flows upward, a ball that passes through a wall. Two benchmarks test this directly. VideoPhy asks humans whether generated clips follow the caption and physical common sense; the best model in that study, CogVideoX-5B, did both in 39.6% of cases. [Certain] Physics-IQ gives the model the start of a real video and asks it to continue; the authors found physical understanding “severely limited” and unrelated to visual realism across the models they tested. [Certain] Simple automatic checks help too: optical-flow statistics (is there motion, is it smooth), tracking a point or an object mask through the clip (does it vanish or split), and comparing generated motion to real motion for the same action class.

Cost: tokens, memory and latency

The cost of one video sample is the cost of one DiT forward pass, times the number of passes, plus the VAE decode. For a transformer with \(L\) layers, width \(d\), MLP width \(d_{ff}\), self-attention over \(N\) tokens and cross-attention to \(N_c\) text tokens, one forward pass costs about

\[ C_{\text{fwd}} \approx L\Big[\underbrace{2N\big(6d^2 + 2 d\, d_{ff}\big)}_{\text{linear layers}} + \underbrace{4 N^2 d}_{\text{self-attention scores}} + \underbrace{4 N N_c d + 4 N_c d^2}_{\text{cross-attention}}\Big] \tag{231.12} \]
Read the symbols
\(6d^2\)
multiply-adds per token for the query, key, value and output projections of self-attention (\(4d^2\)) and the query and output projections of cross-attention (\(2d^2\)); the factor 2 converts multiply-adds to FLOPs
\(2 d\, d_{ff}\)
the two MLP matrices
\(4N^2 d\)
\(QK^\top\) and the weighted sum of values, each \(N^2 d\) multiply-adds
\[ C_{\text{sample}} = K \cdot g \cdot C_{\text{fwd}} + C_{\text{VAE}}, \qquad \frac{\text{attention}}{\text{linear}} = \frac{N}{3d + d_{ff}} \tag{231.13} \]

Here \(K\) is the number of sampling steps and \(g = 2\) with classifier-free guidance (1 without). The ratio on the right says attention dominates once \(N\) is larger than \(3d + d_{ff}\). For Wan 2.1-14B (\(d = 5{,}120\), \(d_{ff} = 13{,}824\), \(L = 40\), 40 heads [Certain]) that threshold is 29,184 tokens, so every 480p and 720p clip is in the attention-dominated regime.

Worked example: the FLOPs of one 5-second 720p sample with Wan 2.1-14B

\(N = 75{,}600\), \(N_c = 512\). Per forward pass, by (231.12): linear and cross-attention terms \(1.84 \times 10^{15}\) FLOPs, self-attention scores \(4.68 \times 10^{15}\) FLOPs, total \(6.52 \times 10^{15}\) (6.52 PFLOP). Attention is 71.8% of the work; the ratio \(N/(3d + d_{ff}) = 2.59\).

With \(K = 50\) steps and guidance (\(g = 2\)): 100 passes, \(6.52 \times 10^{17}\) FLOPs. At a sustained 1 PFLOP/s that is 652 s, about 11 minutes, before the VAE. At 480p (\(N = 32{,}760\)) the same sample is \(1.68 \times 10^{17}\) FLOPs (168 s at 1 PFLOP/s), with attention 52.4%.

Doubling the length to 10 s at 720p (161 frames, \(N = 147{,}600\)) multiplies the cost by 3.29, not 2, because the attention term is quadratic. Replacing full attention by alternating spatial and temporal layers would cut the 5-second 720p pass from 6.52 to 1.95 PFLOP. The sliding tile attention paper reports that for HunyuanVideo, a 5-second 720p clip takes 945 s end to end, of which 800 s is attention. [Certain]

full 3D attentionalternating spatial / temporal layerscurrent setting
Figure 5. FLOPs for one sample with the shapes of Wan 2.1-14B (40 layers, width 5,120, MLP 13,824, 512 text tokens), from (231.1), (231.12) and (231.13). Frames \(F = 16 \times \text{seconds} + 1\); 1080p uses 1088 × 1920 so the size divides by 16. Time is at a sustained 1 PFLOP/s, an ideal round number, not a measured speed; the VAE is not included. The factorized variant keeps the same linear layers and uses spatial attention in half the layers and temporal attention in the other half.
How to read Figure 5

Keep 720p and move the length: the full-attention curve bends upward (quadratic in \(N\)), the factorized curve is almost straight. Look at the bar: at 256² attention is a small slice; at 720p and above it is most of the pass. Turn CFG off and the cost halves; set the steps to 4 (a distilled model, session 230) and a 720p clip drops from about 11 minutes to under 1 minute at the same throughput. The readout also gives the memory one attention score matrix would need if stored: 11.4 GB per head per layer at 720p, which is why FlashAttention, which never stores it, is mandatory.

Memory

Inference memory has three parts. Weights: 14 B parameters are 28 GB in bf16. Activations: at inference only one layer’s activations are alive at a time, for example \(N \times d \times 2\) bytes = 774 MB for the 720p residual stream, plus the MLP hidden state; attention scores are never stored with FlashAttention. The VAE decoder: it works at full pixel resolution and is often the peak; it is run in temporal chunks (the causal design allows a feature cache) and sometimes in spatial tiles, which can create seams if the tiles do not overlap. [Likely] Training is much harder: the Wan report says the activations of the 14B model at one million tokens and micro-batch 1 can exceed 8 TB, and that attention can take up to 95% of training time at that length. [Certain]

The levers that cut cost

  • Fewer steps. Distillation and consistency models (session 230) take 50 steps to 4 to 8; CausVid distills 50 steps into 4. [Certain] Guidance distillation removes the second pass.
  • Sparse attention. Sliding tile attention keeps local 3D windows; it cut HunyuanVideo from 945 s to 685 s with no training, and to 268 s with fine-tuning, at a 0.09% VBench drop. [Certain]
  • Caching across steps. Neighboring sampling steps produce similar features, so some can be reused; Wan reports a 1.62× speedup from a diffusion cache. [Certain]
  • Lower precision. FP8 matrix multiplies and 8-bit attention; Wan reports 1.13× from FP8 GEMMs and more than 1.27× from 8-bit FlashAttention. [Certain]
  • Context (sequence) parallelism. Split the \(N\) tokens across GPUs with Ulysses (all-to-all over heads) and ring attention (pass key-value blocks around a ring); Wan combines both and reports nearly linear speedup. [Certain]
  • Generate small, then upsample. Make a low-resolution or low-frame-rate clip, then run spatial and temporal super-resolution, as Imagen Video did with a cascade of models. [Certain] The cost moves to cheaper stages; the risk is losing consistency between stages.

Video DiT pieces in PyTorch

A complete, deterministic script. It counts tokens with (231.1), implements 3D RoPE and checks that scores depend only on offsets, builds one self-attention layer with full, spatial or temporal attention and measures with gradients which input tokens can reach one output token, and checks the FD bias formula (231.9). It runs in a few seconds on a CPU.

import math, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
torch.set_default_dtype(torch.float64)

# ---------- 1. token count of a video latent (231.1) ----------
def video_tokens(frames, H, W, ct=4, cs=8, p=2):
    T = 1 + (frames - 1) // ct              # causal VAE: first frame kept alone
    return T, H // cs // p, W // cs // p

for name, (fr, H, W) in {"480p 5 s": (81, 480, 832), "720p 5 s": (81, 720, 1280)}.items():
    T, h, w = video_tokens(fr, H, W)
    print(f"{name}: latent frames {T}, patch grid {h}x{w}, tokens {T*h*w}")

# ---------- 2. 3D rotary position embedding (231.3) ----------
def rope_1d(x, pos, base=10000.0):
    # x: (..., n, d) with d even; pos: (n,) integer positions on one axis
    d = x.shape[-1]
    inv = 1.0 / base ** (torch.arange(0, d, 2, dtype=x.dtype) / d)
    ang = pos[:, None].to(x.dtype) * inv[None, :]             # (n, d/2)
    cos, sin = ang.cos(), ang.sin()
    x1, x2 = x[..., 0::2], x[..., 1::2]
    out = torch.stack([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1)
    return out.flatten(-2)

def rope_3d(x, coords, split):
    # split the head dimension into a time part, a height part and a width part
    parts = torch.split(x, split, dim=-1)
    return torch.cat([rope_1d(p, coords[:, i]) for i, p in enumerate(parts)], dim=-1)

def grid(T, h, w):
    t, y, x = torch.meshgrid(torch.arange(T), torch.arange(h), torch.arange(w), indexing="ij")
    return torch.stack([t, y, x], -1).reshape(-1, 3)          # (T*h*w, 3)

# ---------- 3. one video self-attention layer: full 3D or factorized ----------
class VideoSelfAttention(nn.Module):
    def __init__(self, dim, heads, split, mode):
        super().__init__()
        self.h, self.dh, self.split, self.mode = heads, dim // heads, split, mode
        self.qkv, self.out = nn.Linear(dim, 3 * dim), nn.Linear(dim, dim)
    def forward(self, x, T, h, w):                            # x: (B, T*h*w, dim)
        B, N, D = x.shape
        q, k, v = self.qkv(x).view(B, N, 3, self.h, self.dh).unbind(2)
        c = grid(T, h, w)
        q, k = rope_3d(q.transpose(1, 2), c, self.split), rope_3d(k.transpose(1, 2), c, self.split)
        v = v.transpose(1, 2)                                 # (B, heads, N, dh)
        if self.mode == "full":                               # every token sees every token
            o = F.scaled_dot_product_attention(q, k, v)
        else:
            S = h * w
            r = lambda z: z.reshape(B, self.h, T, S, self.dh)
            q, k, v = r(q), r(k), r(v)
            if self.mode == "spatial":                        # attend inside the same frame
                o = F.scaled_dot_product_attention(q, k, v)
            else:                                             # "temporal": same position, all frames
                tr = lambda z: z.transpose(2, 3)
                o = tr(F.scaled_dot_product_attention(tr(q), tr(k), tr(v)))
            o = o.reshape(B, self.h, N, self.dh)
        return self.out(o.transpose(1, 2).reshape(B, N, D))

# ---------- 4. checks ----------
T, h, w, dim, heads = 6, 4, 4, 48, 2                         # head dim 24 = 8 (t) + 8 (h) + 8 (w)
split = [8, 8, 8]
x = torch.randn(1, T * h * w, dim)

# (a) RoPE scores depend only on the offset between two tokens
qv, kv = torch.randn(24), torch.randn(24)
def score(pq, pk):
    a = rope_3d(qv[None], torch.tensor([pq]), split)
    b = rope_3d(kv[None], torch.tensor([pk]), split)
    return float((a * b).sum())
s1, s2 = score([1, 2, 0], [4, 3, 3]), score([2, 0, 1], [5, 1, 4])  # same offset (3, 1, 3)
print(f"RoPE score, offset (3,1,3) at two places: {s1:.6f} {s2:.6f}")

# (b) how many input tokens can change the output token at (t=0, y=0, x=0)?
def reach(modes):
    layers = [VideoSelfAttention(dim, heads, split, m) for m in modes]
    xi = x.clone().requires_grad_(True)
    y = xi
    for L in layers:
        y = y + L(y, T, h, w)                                 # residual connection
    y[0, 0].sum().backward()
    return int((xi.grad[0].abs().sum(-1) > 1e-12).sum())
for modes in [["spatial"], ["temporal"], ["spatial", "temporal"], ["full"]]:
    print(f"{'+'.join(modes):17s}: {reach(modes):3d} of {T*h*w} input tokens reach token (0,0,0)")

# (c) the size of the score matrix, per head
N, S = T * h * w, h * w
print("score entries: full", N * N, " spatial", T * S * S, " temporal", S * T * T)

# ---------- 5. Frechet distance (231.8) and its small-sample bias (231.9) ----------
def sqrtm_psd(A):
    e, V = torch.linalg.eigh(A)
    return (V * e.clamp(min=0).sqrt()) @ V.T
def frechet(X, Y):
    m1, m2 = X.mean(0), Y.mean(0)
    C1, C2 = torch.cov(X.T), torch.cov(Y.T)
    s = sqrtm_psd(C1)
    tr = torch.linalg.eigvalsh(s @ C2 @ s).clamp(min=0).sqrt().sum()
    return float(((m1 - m2) ** 2).sum() + C1.trace() + C2.trace() - 2 * tr)
g = torch.Generator().manual_seed(1)
for d, n in [(16, 256), (16, 2048), (32, 256)]:
    fd = sum(frechet(torch.randn(n, d, generator=g), torch.randn(n, d, generator=g)) for _ in range(20)) / 20
    print(f"same distribution, d={d}, n={n}: mean FD {fd:.3f}   d(d+5)/(2n) = {d*(d+5)/(2*n):.3f}")

On PyTorch 2.8 (CPU) it prints:

480p 5 s: latent frames 21, patch grid 30x52, tokens 32760
720p 5 s: latent frames 21, patch grid 45x80, tokens 75600
RoPE score, offset (3,1,3) at two places: 2.452681 2.452681
spatial          :  16 of 96 input tokens reach token (0,0,0)
temporal         :   6 of 96 input tokens reach token (0,0,0)
spatial+temporal :  96 of 96 input tokens reach token (0,0,0)
full             :  96 of 96 input tokens reach token (0,0,0)
score entries: full 9216  spatial 1536  temporal 576
same distribution, d=16, n=256: mean FD 0.674   d(d+5)/(2n) = 0.656
same distribution, d=16, n=2048: mean FD 0.078   d(d+5)/(2n) = 0.082
same distribution, d=32, n=256: mean FD 2.299   d(d+5)/(2n) = 2.312

The two RoPE scores are equal: only the offset matters. A spatial layer alone never mixes frames, a temporal layer alone never mixes positions, and the pair reaches all 96 tokens in two hops, the same reach as one full layer at 9,216 / (1,536 + 576) = 4.4 times the score entries in this tiny clip (21 times at 720p). The FD of two samples of the same distribution matches the bias formula. Other versions or hardware may print slightly different last digits.

Where this is used

SettingWhat is usedWhat matters
Text-to-video and image-to-video productsLatent video DiT with full 3D attention, 50 steps or a distilled few-step modelLatency and cost per clip; identity stability; prompt following
Synthetic data for video understandingGenerated clips of rare actions or events to train action recognition and key-moment detectorsMotion realism and label correctness matter more than per-frame beauty; check with a held-out real test set [Likely]
Video retrieval and Video-RAGThe same space-time patch encoders and 3D RoPE appear in video encoders; generated clips can stress-test retrievalGenerated near-duplicates can pollute indexes; video fingerprinting must detect them
Video editing and inpaintingInvert a real clip to noise (or condition on its latent), change the prompt or mask a region, sample againTemporal consistency of the edit across frames; reconstruction-guidance style conditioning
Segmentation and trackingPoint and mask tracking used as an automatic check of object permanence in generated clipsObjects that split, vanish or merge are caught without human raters
World models for robots and drivingAction-conditioned or autoregressive video models (Diffusion Forcing, causal DiTs)Long rollouts without drift; physical plausibility; real-time speed
Model evaluation and selectionFVD at a fixed sample size, VBench dimensions, pairwise human preference with EloSame \(n\), same prompts, report confidence intervals; never trust one number

Common mistakes

  • Forgetting the causal first frame. Asking for 80 frames instead of 81 (\(1 + 4k\)) leads to padding, dropped frames or a shape error. Count tokens with (231.1).
  • Estimating cost with \(2NP\) only. The “2 × tokens × parameters” rule ignores the \(4N^2 d\) attention term, which is 72% of the work at 720p. Use (231.12).
  • Comparing FVD across different sample sizes, resolutions, frame counts or feature extractors. The bias (231.9) alone can exceed the difference between models.
  • Trusting FVD or per-frame metrics for motion. FVD is biased toward frame quality; a static clip can score well. Always add motion checks (dynamic degree, flow) and human review.
  • Reading consistency scores without motion scores. A frozen video gets perfect subject consistency and zero flicker. Report dynamic degree next to them.
  • Extending clips with the replacement method only. Without reconstruction guidance or a model trained for conditioning, the new chunk does not match the old one, and errors accumulate over many chunks.
  • Decoding with tiles and no overlap. Spatial tiles of the VAE decoder without overlap and blending show visible seams; temporal chunks without the causal cache show jumps at chunk borders. [Likely]
  • Treating realism as physics. Physics-IQ found physical understanding unrelated to visual realism. Test physics separately.

Interview questions

How many tokens does a 5-second, 16 fps, 720 × 1280 clip become in a Wan-style video DiT?basic
81 frames; the causal 3D VAE compresses time by 4 (first frame alone) and space by 8: \(21 \times 90 \times 160\) latent with 16 channels. The \(1 \times 2 \times 2\) patchify gives \(21 \times 45 \times 80 = 75{,}600\) tokens, 18.5 times a \(1024^2\) image (4,096 tokens).
Why does a 3D VAE use a causal design with the first frame encoded alone?core
Causality (each latent frame depends only on current and past frames) allows encoding and decoding in temporal chunks with a feature cache, which bounds memory for long videos. Encoding the first frame alone makes a single image a valid one-frame video, so one VAE and one DiT can be trained on images and videos together, and image-to-video can encode the given frame exactly as the model expects. The cost is frame counts of the form \(1 + 4k\).
Full 3D attention or factorized spatial/temporal attention: compare cost and quality.core
Full attention has \(N^2 = T^2 S^2\) score entries per layer; one spatial plus one temporal layer has \(T S^2 + S T^2\), about \(T\) times fewer (21 times at 720p). Factorized attention connects any two tokens only in two hops along fixed axes, so a moving object must be tracked through “same position” paths that may contain background. Full attention matches content wherever it is, which helps motion and identity. Large recent models (CogVideoX, HunyuanVideo, Movie Gen, Wan) chose full attention and pay with FlashAttention, context parallelism and sparse or windowed attention at inference.
Why does running an image diffusion model frame by frame flicker, even with the same seed?core
The sampler decides content at high noise. Each frame’s denoiser sees only its own frame, so each frame makes its own decision; even shared initial noise does not force the same decision once the inputs differ slightly. Each frame is a valid sample of the per-frame marginal, but the joint distribution is wrong. In the toy of Figure 3, 16 independent frames give about 7.5 identity switches per clip; a denoiser that sees the whole clip gives 0. The fix is joint denoising with attention across frames, plus conditioning on a reference frame.
What is 3D RoPE, and why is it a good fit for video?core
The head dimension is split into three groups, rotated by angles proportional to the frame index, row and column (231.3). The query-key dot product then depends only on the 3D offset, so the same relative motion looks the same anywhere in the clip and at any absolute time. It adds no parameters and extends naturally to other lengths and resolutions. Wan splits 128 head dims as 44/42/42; HunyuanVideo as 16/56/56.
What does FVD measure, and what are its weaknesses?core
FVD fits Gaussians to I3D (Kinetics-400) features of real and generated clips and computes the Fréchet distance (231.8). Weaknesses: it is biased upward at small sample sizes, roughly \(d(d+5)/(2n)\) for unit-variance features, so only same-\(n\) comparisons are valid; it is biased toward per-frame quality (content bias), so static or temporally broken clips can score well; I3D was trained on short low-resolution clips of human actions, so it may not reflect long, high-resolution or non-human content; and one number hides which aspect failed.
Estimate the FLOPs and the share of attention for one 720p, 5-second sample of a 14B video DiT.advanced
With \(L = 40\), \(d = 5{,}120\), \(d_{ff} = 13{,}824\), \(N = 75{,}600\), 512 text tokens: linear plus cross-attention \(\approx 1.84\) PFLOP and self-attention scores \(4N^2 d L \approx 4.68\) PFLOP per pass, total 6.52 PFLOP, 72% attention (\(N/(3d + d_{ff}) = 2.59\)). 50 steps × 2 for CFG = 100 passes = \(6.5 \times 10^{17}\) FLOPs, about 11 minutes at a sustained 1 PFLOP/s, plus the VAE.
You must cut latency of a video model by 10×. What do you try, in what order?advanced
First the number of passes: guidance distillation (2× fewer passes) and step distillation or consistency training from 50 to 4 to 8 steps (6 to 12×). Then attention, which is most of each pass: sliding-window or sliding-tile attention (training-free first, then fine-tuned), and caching of features across neighboring steps. Then systems: FP8 matrix multiplies, 8-bit attention, context parallelism across GPUs (Ulysses plus ring attention) for latency rather than throughput. Consider generating at lower resolution and upsampling. Re-check temporal metrics and human preference after each change, since speedups often hurt motion before they hurt single frames.
How do you generate a video longer than the training length, and what goes wrong?advanced
Autoregressive extension: generate a chunk, then generate the next chunk conditioned on the last frames of the previous one. The naive replacement method gives chunks that look fine alone but do not connect; reconstruction guidance (231.7) adds a gradient that makes the model’s reconstruction of the known frames match them. Over many chunks errors accumulate (color drift, identity drift). Remedies: train with per-frame noise levels (Diffusion Forcing), causal models trained to roll out, conditioning on a reference frame or identity embedding, and overlapping windows.
A new model has better FVD but people prefer the old one. How can that happen?core
FVD may have been computed with a different sample size or preprocessing; the new model may produce sharper frames with less or worse motion, which FVD rewards because of its content bias; it may be closer to the reference set’s statistics but worse at following prompts, which FVD does not measure at all; or human raters focus on artifacts (hands, faces, physics) that move a 400-dim action classifier’s features very little. Check VBench-style dimensions, especially dynamic degree, and run a pairwise study with enough prompts.
Model A has an Elo rating 100 points above model B. What share of pairwise votes should A win?basic
\(1/(1 + 10^{-100/400}) = 0.640\): about 64%. At 200 points it is 76%. Report confidence intervals; ratings from few votes move a lot.

Exercises

  1. Count the tokens of a 10-second, 16 fps clip at 480 × 832 with a 4 × 8 × 8 VAE and 1 × 2 × 2 patches. How many times more is this than the 5-second clip?
  2. For \(T = 21\) and \(S = 1{,}560\) (480p), compute the score entries of full attention and of one spatial plus one temporal layer, and their ratio.
  3. Using (231.13), at what number of tokens does attention start to dominate for Wan 2.1-1.3B (\(d = 1{,}536\), \(d_{ff} = 8{,}960\))? Is a 5-second 480p clip above it?
  4. Real features \(\mathcal{N}(1, 1)\), generated features \(\mathcal{N}(1, 0.5^2)\), one dimension. Compute the Fréchet distance. Which term contributes?
  5. You compute FVD with 400-dimensional features and unit-variance features (an idealization). Using (231.9), how large is the bias at \(n = 2{,}048\) per side? How many clips per side would you need to bring it below 5?
  6. In Figure 3, set \(w = 0\). Explain why the expected number of identity switches is 7.5 and why no clip is consistent among 300.
Answerscheck
1. 161 frames, \(T = 1 + 160/4 = 41\); \(41 \times 30 \times 52 = 63{,}960\) tokens, 1.95 times the 32,760 of the 5-second clip (the first frame is shared). 2. Full: \((21 \times 1{,}560)^2 = 32{,}760^2 = 1.073 \times 10^9\). Factorized: \(21 \times 1{,}560^2 + 1{,}560 \times 21^2 = 51{,}105{,}600 + 687{,}960 = 51{,}793{,}560\). Ratio 20.7. 3. \(3d + d_{ff} = 4{,}608 + 8{,}960 = 13{,}568\) tokens; 32,760 is 2.4 times above it, so attention dominates (the 1.3B model is in the attention regime even at 480p). 4. \((1 - 1)^2 + (1 - 0.5)^2 = 0.25\); only the spread term contributes: the generator is not diverse enough. 5. \(400 \times 405 / (2 \times 2{,}048) = 162{,}000 / 4{,}096 = 39.6\). Need \(n \gt 162{,}000/10 = 16{,}200\) clips per side. Real I3D features are not unit-variance, so the real bias differs, but the lesson holds: fix \(n\). 6. With \(w = 0\) each frame picks object A or B with probability 1/2 independently, so each of the 15 neighbor pairs differs with probability 1/2: \(15 \times 0.5 = 7.5\). The chance that all 16 frames agree is \(2 \times 2^{-16} = 3.1 \times 10^{-5}\), so among 300 clips we expect 0.009 consistent clips.

Key takeaways

  • Modern video generators are latent DiTs: a causal 3D VAE (4× time, 8× space, 16 channels) plus \(1 \times 2 \times 2\) patches turns a 5-second 720p clip into 75,600 tokens (231.1).
  • Full 3D attention lets every token see every other token; factorized attention is about \(T\) times cheaper (231.2) but connects frames only along fixed axes. Large recent models use full attention with 3D RoPE (231.3).
  • Temporal coherence comes from denoising all frames jointly: when frames cannot see each other at high noise, they make independent content decisions and flicker (231.6, Figure 3). Joint image-video training, frame conditioning and reconstruction guidance (231.7) help.
  • FVD (231.8) is the standard metric, but it is biased by sample size (231.9) and toward single-frame quality. Pair it with VBench dimensions (231.10), human preference (231.11) and physics tests.
  • Cost: attention dominates once \(N \gt 3d + d_{ff}\) (231.13). A 720p 5-second sample of a 14B model is about \(6.5 \times 10^{17}\) FLOPs; steps, CFG, sparse attention, caching and parallelism are the levers.

Key formulas

\[ N = \Big(1 + \frac{F-1}{c_t}\Big)\frac{H}{c_s p}\cdot\frac{W}{c_s p}, \qquad \text{full: } T^2 S^2 \ \text{ vs. factorized: } T S^2 + S T^2 \]
\[ C_{\text{fwd}} \approx L\big[2N(6d^2 + 2 d\,d_{ff}) + 4N^2 d\big], \qquad C_{\text{sample}} = K g\, C_{\text{fwd}} + C_{\text{VAE}} \]
\[ \text{FVD} = \lVert \boldsymbol{\mu}_r - \boldsymbol{\mu}_g \rVert^2 + \operatorname{tr}\big(\Sigma_r + \Sigma_g - 2(\Sigma_r\Sigma_g)^{1/2}\big), \qquad \E[\widehat{\text{FD}}] \approx \text{FD} + \frac{d(d+5)}{2n} \]

References for this session

  1. Ho et al. (2022). Video Diffusion Models
  2. Unterthiner et al. (2018). Towards Accurate Generative Models of Video: A New Metric & Challenges
  3. Peebles & Xie (2022). Scalable Diffusion Models with Transformers
  4. Ho et al. (2022). Video Diffusion Models
  5. Unterthiner et al. (2018). Towards Accurate Generative Models of Video: A New Metric & Challenges
  6. Peebles & Xie (2022). Scalable Diffusion Models with Transformers
  7. Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models
  8. Liu et al. (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
  9. Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
  10. Ho & Salimans (2022). Classifier-Free Diffusion Guidance
  11. Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
  12. Arnab et al. (2021). ViViT: A Video Vision Transformer
  13. Bertasius et al. (2021). Is Space-Time Attention All You Need for Video Understanding?
  14. Carreira & Zisserman (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
  15. Caron et al. (2021). Emerging Properties in Self-Supervised Vision Transformers
  16. Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  17. Liu et al. (2023). Ring Attention with Blockwise Transformers for Near-Infinite Context
  18. Jacobs et al. (2023). DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
  19. Salimans & Ho (2022). Progressive Distillation for Fast Sampling of Diffusion Models
  20. Song et al. (2023). Consistency Models
  21. Wan Team (Alibaba) (2025). Wan: Open and Advanced Large-Scale Video Generative Models
  22. Yang et al. (2024). CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
  23. Kong et al. (2024). HunyuanVideo: A Systematic Framework For Large Video Generative Models
  24. Polyak et al. (2024). Movie Gen: A Cast of Media Foundation Models
  25. Ma et al. (2024). Latte: Latent Diffusion Transformer for Video Generation
  26. Blattmann et al. (2023). Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  27. Blattmann et al. (2023). Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
  28. Bar-Tal et al. (2024). Lumiere: A Space-Time Diffusion Model for Video Generation
  29. Ho et al. (2022). Imagen Video: High Definition Video Generation with Diffusion Models
  30. Brooks et al. (OpenAI) (2024). Video generation models as world simulators (technical report)
  31. Huang et al. (2023). VBench: Comprehensive Benchmark Suite for Video Generative Models
  32. Ge et al. (2024). On the Content Bias in Fréchet Video Distance
  33. Motamed et al. (2025). Do generative video models understand physical principles?
  34. Bansal et al. (2024). VideoPhy: Evaluating Physical Commonsense for Video Generation
  35. Zhang et al. (2025). Fast Video Generation with Sliding Tile Attention
  36. Chen et al. (2024). Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion
  37. Yin et al. (2024). From Slow Bidirectional to Fast Autoregressive Video Diffusion Models
  38. Teed & Deng (2020). RAFT: Recurrent All-Pairs Field Transforms for Optical Flow