Flow matching and rectified flow
Flow matching trains a network to predict a velocity: the direction and speed that move a noise sample toward data. Training is a simple regression on straight lines between noise and data, and sampling is solving an ordinary differential equation. Rectified flow makes the paths straighter so fewer steps are needed, and this recipe, with a better choice of training timesteps, is what Stable Diffusion 3 and FLUX use.
What you will learn
- What a velocity field and an ODE are, and how an Euler step moves a sample from noise to data
- The straight path x_t = (1 − t) x + t ε, its constant velocity ε − x, and why the training target is that simple
- Why the flow matching loss on single pairs has the same gradient as the loss on the true (averaged) velocity, and what the network really learns
- How flow matching relates to diffusion: x-, ε- and v-prediction are one model with different loss weights
- Rectified flow and reflow: why learned paths curve, how re-pairing noise with its own output straightens them, and the cost in quality
- The SD3 recipe: logit-normal timestep sampling, the resolution shift, and the MM-DiT transformer; FLUX.1 as a 12B rectified flow transformer
- Common mistakes, from the flipped time convention between papers to one-step sampling without reflow
Why this matters
Diffusion models generate by removing noise in many small steps. Flow matching keeps the same idea (start from noise, end at data) but asks the network a simpler question: “at this point and this time, in which direction and how fast should the sample move?” The answer is a velocity. Training is plain regression, and sampling is following the velocity like a boat following a river current.
This is not a side topic. Stable Diffusion 3 (Esser et al., 2024) is trained with rectified flow, the simplest kind of flow matching, and its paper reports that rectified flow with a better choice of training timesteps beat the established diffusion formulations in a large comparison [Certain] (SD3 paper). FLUX.1 is described by its authors as a “12 billion parameter rectified flow transformer” [Certain] (model card). Meta’s Movie Gen video model, a 30B-parameter transformer, is also trained with flow matching [Likely]. If you read about a modern image or video generator, you will meet the words “velocity”, “rectified flow” and “logit-normal”.
The session builds four ideas, each a common interview question:
- A velocity field moves noise to data. A network \(\mathbf{v}_\theta(\mathbf{x}, t)\) defines an ordinary differential equation (ODE); solving it from \(t = 1\) (noise) to \(t = 0\) (data) generates a sample (formula 229.1).
- The training loss is simple regression. Pick a data point, a noise sample and a time, mix them on a straight line, and ask the network for the line’s velocity \(\epsilon - \mathbf{x}\) (formula 229.8).
- Straight paths are cheap to follow. The learned paths are not straight, but reflow makes them straighter, so a few steps are enough (Figures 2 to 4).
- The SD3 recipe: sample training timesteps from a logit-normal distribution, shift them for high resolution, and use a two-stream transformer (MM-DiT).
We build on session 225 (forward noising, \(\epsilon\)-prediction), session 226 (the probability-flow ODE and DDIM, which are already deterministic flows) and session 228 (the latent space and the DiT backbone). The next session, session 230, pushes the step count down to one or two.
The idea: a velocity field that moves noise to data
Think of every point \(\mathbf{x}\) in space (an image, or a latent) as having an arrow attached: a velocity \(\mathbf{v}(\mathbf{x}, t)\) that says how the point should move at time \(t\). A collection of such arrows for all points is a velocity field. If you drop a particle somewhere and let it follow the arrows, it traces a path. Writing this as an equation gives an ordinary differential equation (ODE):
- Read the symbols
- \(t\)
- time, from 1 (pure noise) down to 0 (data). We use the convention of SD3 and of the diffusion sessions: \(t = 0\) is data.
- \(\mathbf{x}_t\)
- the sample at time \(t\): a vector, for example a flattened latent image
- \(\mathbf{v}_\theta\)
- the network: input a point and a time, output a vector of the same size (the velocity)
- \(\mathcal{N}(\mathbf{0}, \mathbf{I})\)
- a standard Gaussian: where every sample starts
A model of this kind is called a continuous normalizing flow (CNF). CNFs existed before 2022, but they were trained by simulating the ODE during training, which is slow. Flow matching (Lipman et al., 2022) trains them without simulation: no ODE is solved during training, just one network call per example, like diffusion.
To generate, we solve the ODE backward from \(t = 1\) to \(t = 0\). The simplest solver is Euler’s method: take a small time step \(\Delta\) and move along the current arrow:
The minus sign is there because time goes down. Each step costs one network call, so the number of steps is the cost of generation. Euler is exact when the path is a straight line traveled at constant speed: then one step of size 1 lands exactly on the end. That single fact explains most of this session.
Different from diffusion sampling with DDPM (session 225), there is no fresh noise added during sampling: the same starting noise always gives the same output. In session 226 you met the probability-flow ODE and DDIM, which are also deterministic. Flow matching starts from that ODE view and makes it the whole model.
The path: a straight line between noise and data
To train a velocity field, we first need to say which path a sample should take. Flow matching builds the answer from the simplest possible piece. Take one data point \(\mathbf{x}\) and one noise sample \(\epsilon\), and connect them with a straight line:
At \(t = 0\) this is the data point, at \(t = 1\) it is the noise. In between it is a mix, exactly like the noisy image \(\sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon\) of DDPM, only with different weights: here the two weights add up to 1 (and the variance dips in the middle), while in DDPM their squares add up to 1. The velocity along this line is its time derivative, and it does not depend on \(t\):
This is the conditional velocity: the velocity of the path for one given pair \((\mathbf{x}, \epsilon)\). It is constant, so this path is straight and traveled at constant speed. This kind of path is what Liu et al. (2022) call rectified flow and what Lipman et al. (2022) call the optimal-transport (OT) conditional path. Albergo and Vanden-Eijnden (2022) proposed the same kind of construction under the name stochastic interpolants. Three groups, the same year, the same idea.
One number, so you can check by hand. Data \(x = 2\), noise \(\epsilon = -1\), time \(t = 0.25\).
The point on the line: \(x_t = 0.75 \times 2 + 0.25 \times (-1) = 1.25\). The target velocity: \(\epsilon - x = -1 - 2 = -3\). If the network outputs \(\hat v = -2.5\), the squared error is \((-2.5 - (-3))^2 = 0.25\).
From a velocity you can read off both endpoints: \(\hat x = x_t - t \hat v\) and \(\hat\epsilon = x_t + (1 - t)\hat v\). With the true \(v = -3\): \(\hat x = 1.25 + 0.75 = 2.0\) and \(\hat\epsilon = 1.25 - 2.25 = -1.0\). Both are exact, as they should be (formula 229.11 below).
Many pairs give many lines, and together they move a whole distribution. At each time \(t\), the points \(\mathbf{x}_t\) of all pairs form a distribution \(p_t\): the data at \(t = 0\), the Gaussian at \(t = 1\), and a blend in between. This family \(p_t\) is called the probability path. A velocity field \(\mathbf{u}_t\) moves the distribution along the path if mass is conserved, which is the continuity equation from physics:
In words: the density at a point changes only because probability flows in or out. You do not need to compute with this equation; it is the guarantee behind the method. If the network learns the right \(\mathbf{u}_t\), the ODE (229.1) carries Gaussian noise exactly onto the data distribution.
The flow matching loss: why simple regression works
Here is the problem. Many straight lines pass through the same point \(\mathbf{z}\) at the same time \(t\): different pairs \((\mathbf{x}, \epsilon)\) can produce the same \(\mathbf{x}_t\). Each of these lines has a different velocity. But an ODE needs one velocity per point. The right one is the average of the velocities of all lines through that point, weighted by how likely each line is:
This is the marginal velocity. It is the field that satisfies the continuity equation (229.5) for the path \(p_t\) [Certain] (Lipman et al., Theorem 1; Liu et al.). The natural loss would be to match it directly, the flow matching (FM) loss:
We cannot compute \(\mathbf{u}_t(\mathbf{z})\): it is an average over the whole dataset. The trick is to regress on the velocity of one line instead, the conditional flow matching (CFM) loss:
- Read the symbols
- \(t\)
- a random time in \([0, 1]\); uniform in the basic method, logit-normal in SD3 (formula 229.16)
- \(\mathbf{x} \sim p_\text{data}\)
- a random training example (in latent diffusion, a VAE latent)
- \(\epsilon\)
- fresh Gaussian noise of the same shape
- \(\epsilon - \mathbf{x}\)
- the target: the velocity of the straight line through this pair
The key result: the two losses differ by a constant that does not depend on \(\theta\), so they have the same gradient:
Why? This is the same fact you use whenever you train with mean squared error. For a fixed input, the prediction that minimizes \(\E\lVert \mathbf{v} - \mathbf{y}\rVert^2\) over noisy targets \(\mathbf{y}\) is their conditional mean \(\E[\mathbf{y} \mid \text{input}]\). Expand the square: \(\lVert \mathbf{v} - \mathbf{y}\rVert^2 = \lVert \mathbf{v} - \E[\mathbf{y}]\rVert^2 + \lVert \E[\mathbf{y}] - \mathbf{y}\rVert^2 + \) a cross term whose average is zero. The second term is the constant \(C\): the spread of the targets of all lines through the same point. So a network trained on noisy single-pair targets converges to the average velocity (229.6) [Certain] (Lipman et al., Theorem 2).
Data is \(+1\) or \(-1\) with equal probability; noise is \(\mathcal{N}(0, 1)\). Look at the point \(z = 0.2\) at time \(t = 0.5\). Two lines pass through it:
From \(x = +1\): the noise must be \(\epsilon = (z - (1 - t)x)/t = (0.2 - 0.5)/0.5 = -0.6\), so the target is \(\epsilon - x = -1.6\). From \(x = -1\): \(\epsilon = (0.2 + 0.5)/0.5 = 1.4\), target \(1.4 + 1 = 2.4\).
How likely is each line? Given \(x\), the point \(x_t\) is Gaussian with mean \((1 - t)x\) and standard deviation \(t\). This gives weights \(0.690\) for \(x = +1\) and \(0.310\) for \(x = -1\). The marginal velocity is the weighted average: \(0.690 \times (-1.6) + 0.310 \times 2.4 = -0.360\).
For two-point data there is a closed form: the best guess of the data is \(\hat x = \tanh\big((1 - t) z / t^2\big) = \tanh(0.4) = 0.380\), and \(u = (z - \hat x)/t = (0.2 - 0.380)/0.5 = -0.360\). Same number. The network is trained on targets \(-1.6\) and \(2.4\) but learns \(-0.360\), a velocity that no single training line has.
The last line of that example is the general formula for our straight path. Since \(\epsilon = (\mathbf{x}_t - (1 - t)\mathbf{x})/t\), the marginal velocity can be written as \(\mathbf{u}_t(\mathbf{z}) = (\mathbf{z} - \hat{\mathbf{x}}(\mathbf{z}, t))/t\), where \(\hat{\mathbf{x}} = \E[\mathbf{x} \mid \mathbf{x}_t = \mathbf{z}]\) is the best guess of the clean data. For data that is a mixture of Gaussians, \(\hat{\mathbf{x}}\) has a closed form, so we can compute the exact optimal velocity field. Figures 1 to 4 do this for 2-D data made of 8 Gaussian blobs on a circle (radius 2, standard deviation 0.15). No trained network is involved in Figure 1: it shows what a perfectly trained network would learn.
How to read Figure 1
Press Play, or move the time slider from 1 to 0 yourself. The orange dots slide along straight lines, but the lines cross everywhere: at a crossing, two training targets disagree, and the network can only learn their average. The dark dots follow that average. Their paths never cross, because an ODE has one velocity per point. Notice that many of these paths first move inward, toward the center of the data, and only later turn toward one blob. Early on (\(t\) near 1) the noisy point says almost nothing about which blob it will reach, so the best guess \(\hat{\mathbf{x}}\) is near the data mean. As \(t\) falls, the guess commits to one blob and the path bends. The training paths are straight; the learned paths are not.
Training is now easy to write down. It looks almost exactly like DDPM training from session 225: only the mixing weights and the target change.
- input: dataset (images or VAE latents), network \(\mathbf{v}_\theta\), timestep distribution \(\pi(t)\)
- repeat
- sample a batch \(\mathbf{x}\) from the data, \(\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) of the same shape, \(t \sim \pi(t)\) // uniform, or logit-normal as in SD3
- \(\mathbf{x}_t \leftarrow (1 - t)\,\mathbf{x} + t\,\epsilon\) // formula (229.3)
- \(L \leftarrow \lVert \mathbf{v}_\theta(\mathbf{x}_t, t) - (\epsilon - \mathbf{x}) \rVert^2\), averaged over the batch // formula (229.8)
- take an AdamW step on \(\nabla_\theta L\)
- until samples are good (not until the loss is near zero: it cannot be)
Sampling: solve the ODE
Sampling is Euler’s method (229.2), run from \(t = 1\) to \(t = 0\):
- input: trained \(\mathbf{v}_\theta\), number of steps \(N\), optional time grid \(1 = t_0 \gt t_1 \gt \dots \gt t_N = 0\) // uniform, or shifted (229.18)
- \(\mathbf{x} \leftarrow \epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\)
- for \(i = 0, \dots, N - 1\):
- \(\mathbf{x} \leftarrow \mathbf{x} + (t_{i+1} - t_i)\, \mathbf{v}_\theta(\mathbf{x}, t_i)\) // \(t_{i+1} - t_i \lt 0\): we move against the velocity
- return \(\mathbf{x}\) // a latent: decode it with the VAE in latent models
With classifier-free guidance (session 227), each step calls the network twice, with and without the prompt, and uses \(\mathbf{v}_\text{uncond} + w(\mathbf{v}_\text{cond} - \mathbf{v}_\text{uncond})\). Higher-order solvers such as Heun (two network calls per step) reduce the error per step; adaptive solvers such as Dormand–Prince choose the step size themselves. Lipman et al. stress that off-the-shelf ODE solvers work well for these models [Certain] (abstract).
Figure 2 runs Algorithm 2 in your browser on the same 8-blob data, one line at a time. The velocity is the exact marginal velocity (229.6) of 1-rectified flow, or the 2-rectified flow that reflow produces (built in Figure 3). Every number in the panel is computed live: the velocity at the tracked sample, the guess \(\hat{\mathbf{x}}\) of formula (229.11), and the update (229.2) with the real values put in.
- input: \(\mathbf{v}_\theta\), \(N\); grid \(t_i = 1 - i/N\)
- \(\mathbf{x} \leftarrow \epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\)
- for \(i = 0, \dots, N - 1\):
- \(\mathbf{v} \leftarrow \mathbf{v}_\theta(\mathbf{x}, t_i)\) // one network call
- \(\mathbf{x} \leftarrow \mathbf{x} + (t_{i+1} - t_i)\,\mathbf{v}\) // (229.2)
- return \(\mathbf{x}\)
How to read Figure 2
Press Play with \(N = 4\) and 1-rectified flow. The tracked sample starts at \(\epsilon = (-0.190, 0.475)\). In the first step, at \(t = 1\), the guess \(\hat{\mathbf{x}}\) is the data mean \((0, 0)\), so \(\mathbf{v} = \epsilon\) and the sample moves a quarter of the way to the center, to \((-0.142, 0.356)\). At \(t = 0.5\) the guess has moved toward the top blob, and in the last step the velocity is large, \((1.297, -3.560)\): the sample jumps to \((-0.606, 1.597)\). The true path ends at \((-0.275, 1.765)\), so the four straight steps miss by 0.372; 79% of all samples land in a blob. With \(N = 16\) the miss is 0.055 and 99% land in a blob. With \(N = 1\) every sample goes to the center: 0%. Now switch to the 2-rectified flow: one step already puts 70% of the samples into blobs, and two steps 81%, because its paths are almost straight.
Why one step gives a blurry average
What does a single Euler step from \(t = 1\) produce? It is \(\mathbf{x} = \epsilon - \mathbf{u}_1(\epsilon)\). At \(t = 1\) the noisy point carries no information about the data, so \(\hat{\mathbf{x}}(\epsilon, 1) = \E[\mathbf{x}]\), the data mean, and \(\mathbf{u}_1(\epsilon) = \epsilon - \E[\mathbf{x}]\). So one step sends every noise sample to the data mean. For images, that is a gray blur. This is the price of the averaging in (229.6): the learned paths curve, and a single straight step cannot follow a curve. Figure 4 shows it: 0% of one-step samples land in a data blob.
Flow matching and diffusion: one family
Is flow matching a new kind of model, or diffusion in different clothes? Both. Lipman et al. define the method for a general family of Gaussian paths, and diffusion paths are members [Certain] (abstract):
Rectified flow is \(a_t = 1 - t\), \(b_t = t\). The variance-preserving diffusion path of DDPM has \(a_t^2 + b_t^2 = 1\), for example \(a_t = \cos(\pi t/2)\), \(b_t = \sin(\pi t/2)\) (the cosine schedule in continuous time). Training any of them with velocity regression gives a deterministic ODE that is the probability-flow ODE of session 226 for that path. What changes between “diffusion” and “flow matching” is (1) the path, (2) what the network outputs, and (3) how the loss weights the different times.
Point (2) is not a real difference, because the three outputs convert into each other. For the straight path, given \(\mathbf{x}_t\) and a predicted velocity \(\hat{\mathbf{v}}\):
Put the true values in and subtract: the errors are tied together by fixed factors, so the three losses are the same loss with different weights over time:
At \(t = 0.5\) the velocity loss is 4 times the \(\epsilon\) loss and 4 times the \(\mathbf{x}\) loss. Near \(t = 1\) (almost pure noise), the velocity loss puts a huge weight on the \(\epsilon\) error; near \(t = 0\), on the \(\mathbf{x}\) error. The SD3 paper compares its formulations in this way, as one loss with different time weightings [Likely].
For completeness, Lipman et al. use the opposite time convention (\(t = 0\) noise, \(t = 1\) data) and add a tiny noise level \(\sigma_\text{min}\) at the data end. In their notation, with \(\mathbf{x}_0\) the noise and \(\mathbf{x}_1\) the data:
With \(\sigma_\text{min} = 0\) and the time flipped (\(t \to 1 - t\)), this is our formula (229.3), and the velocity is ours with the sign flipped. Keep this in mind when you read code: the sign of the velocity depends on the convention.
Rectified flow: making the paths straight
Figure 1 showed the problem: the training lines are straight, but they cross, so the learned flow curves, and curves need many steps. Liu, Gong and Liu (2022) asked: can we pick the pairs so that the lines do not cross? If no two lines cross, there is no averaging, the learned velocity is the velocity of the one line through each point, and the flow follows straight lines. Then one Euler step is exact.
The pairing of noise with data is called a coupling. Basic flow matching uses the independent coupling: any noise with any data. Rectified flow’s answer is reflow: use the trained flow itself to create a better coupling. Run the ODE from each noise sample \(\epsilon\) to its output \(\Phi_k(\epsilon)\), and train a new flow on the pairs \((\Phi_k(\epsilon), \epsilon)\):
- Read the symbols
- \(\Phi_k(\epsilon)\)
- the sample that the \(k\)-th flow produces from noise \(\epsilon\) (solve its ODE from \(t = 1\) to 0)
- \(\mathbf{v}^{(1)}\)
- “1-rectified flow”: plain flow matching with independent pairs
- \(\mathbf{v}^{(2)}\)
- “2-rectified flow”: trained once more on its own pairs
Why does this help? The paths of an ODE never cross. So the new pairs \((\Phi_k(\epsilon), \epsilon)\) are already connected by non-crossing (curved) paths, and the straight lines between them cross much less. Liu et al. prove two properties [Certain] (rectified flow paper): the new coupling still maps the noise distribution exactly onto the same data distribution as the old flow, and the transport cost \(\E\, c(\Phi(\epsilon) - \epsilon)\) does not increase for every convex cost \(c\). They measure how far a flow is from straight with the straightness:
where \(\mathbf{z}_t\) is the path from \(\epsilon\). \(S = 0\) means every path is a straight line at constant speed. Notice that (229.15) on the reflow pairs is just the CFM loss on those pairs: a reflowed model has a much smaller loss floor \(C\). In the PyTorch code below, the loss drops from 1.93 to 0.001 after reflow.
- train \(\mathbf{v}^{(1)}\) on random pairs \((\mathbf{x}, \epsilon)\) // Algorithm 1
- \(\Phi_1(\epsilon_j) \leftarrow\) solve the ODE of \(\mathbf{v}^{(1)}\) from each \(\epsilon_j\)
- new pairs \((\Phi_1(\epsilon_j), \epsilon_j)\), straight lines // (229.14)
- train \(\mathbf{v}^{(2)}\) on them; sample from fresh noise
How to read Figure 3
Compare step 1 and step 3: the same noise points, the same target distribution, but a new partner for each noise point. The partners of step 3 are much closer (the mean squared distance drops by about 7 times) and the lines barely cross. In step 4 the new flow is trained on those lines. The 1-RF velocity is exact here; the 2-RF velocity is estimated by kernel regression on the 1,500 reflow pairs (each query averages the targets of nearby pair lines, bandwidth 0.1), which plays the role of a trained network. The 2-RF paths start from fresh noise points that were not in the training pairs.
Straight paths pay off at sampling time. Figure 4 runs Euler’s method with \(N\) steps for the exact 1-rectified flow and for the 2-rectified flow, from the same 400 noise points.
How to read Figure 4
At \(N = 1\), 1-rectified flow sends every sample to the center (the data mean): 0% in a blob. The 2-rectified flow already puts about three quarters of the samples into blobs with one step. At 3 or 4 steps 1-RF catches up fast, and from about 8 steps it is better: 2-RF stops near 94%, because it can only be as good as the pairs it was trained on and its own fit to them (here the kernel smoothing). This is the honest trade-off of reflow: much better at 1 to 2 steps, a small loss of quality at many steps.
- input: trained flow \(\mathbf{v}^{(k)}\), ODE solver with enough steps for good samples
- draw many noise samples \(\epsilon_1, \dots, \epsilon_M\) // for text-to-image, also a prompt for each
- \(\mathbf{x}_j \leftarrow \Phi_k(\epsilon_j)\) by solving the ODE of \(\mathbf{v}^{(k)}\) // expensive: \(M\) full generations
- train \(\mathbf{v}^{(k+1)}\) with Algorithm 1, but on the fixed pairs \((\mathbf{x}_j, \epsilon_j)\) instead of random pairs // formula (229.14); often initialized from \(\mathbf{v}^{(k)}\)
- optional: distill \(\mathbf{v}^{(k+1)}\) into a one-step model by regressing \(\Phi_{k+1}(\epsilon)\) from \(\epsilon\) directly
Reflow is expensive: you must generate a large synthetic dataset with the teacher, and errors of the teacher are inherited (Figure 4). InstaFlow (Liu et al., 2023) applied reflow plus distillation to Stable Diffusion and turned it into a one-step text-to-image model; the authors report that reflow “plays a critical role in improving the assignment between noise and images” [Certain] (InstaFlow abstract). Most large production models, including SD3, do not run reflow; they train 1-rectified flow and then use other distillation methods to reduce steps [Likely]. Those methods are the subject of session 230.
Rectified-flow transformers: the SD3 recipe
Esser et al. (2024) trained rectified flow at scale for text-to-image (SD3), with three changes that matter: which timesteps to train on, how to adapt to resolution, and the network architecture.
Which timesteps to train on: logit-normal sampling
With uniform \(t\), every noise level gets the same number of training examples. But not all levels are equally hard. Near \(t = 0\) the answer is almost given (little noise to remove); near \(t = 1\) the velocity is almost \(\epsilon - \E[\mathbf{x}]\), easy to learn. The SD3 paper notes that predicting the velocity is hardest for \(t\) in the middle of \([0, 1]\), and puts more training weight there with a logit-normal distribution [Certain] (SD3 paper). Sample \(u \sim \mathcal{N}(m, s^2)\) and set \(t = 1/(1 + e^{-u})\). Its density is:
In the SD3 comparison of many formulations and timestep samplers, rectified flow with logit-normal \(m = 0\), \(s = 1\) had the best average rank [Certain] (SD3 paper). With these values, 72.8% of training timesteps fall in the middle band \(0.25 \le t \le 0.75\), against 50% for uniform, and only 1.4% fall above \(t = 0.9\). Figure 5 lets you change \(m\) and \(s\).
There is a neat way to see what logit-normal means. The signal-to-noise ratio of the straight path is the squared ratio of the signal weight to the noise weight:
So a normal distribution on \(\operatorname{logit}(t)\) is a normal distribution on the log-SNR \(\lambda\): logit-normal sampling with \(m = 0\), \(s = 1\) means \(\lambda \sim \mathcal{N}(0, 4)\), centered where signal and noise are equally strong. Choosing a timestep distribution is choosing which noise levels the model practices on, the same question as the noise schedule in session 225.
Higher resolution needs more noise: the timestep shift
Neighboring pixels of a large image are strongly correlated. At the same \(t\), a 1024×1024 image keeps much more recognizable structure than a 256×256 one, because averaging many correlated noisy pixels removes noise. So a high-resolution model needs to spend more of its time at higher noise. SD3 remaps the timesteps with a shift \(\alpha\):
The paper motivates \(\alpha = \sqrt{m/n}\) for going from \(n\) to \(m\) pixels (that would be 4 from 256² to 1024²), but chose \(\alpha = 3.0\) for 1024×1024 from a human preference study, “both during training and sampling” [Certain] (SD3 paper). The right-hand form shows what the shift does: it adds \(\log\alpha\) to the logit, so it moves a logit-normal distribution from \(m\) to \(m + \log \alpha\), and it divides the SNR at every step by \(\alpha^2\). With \(\alpha = 3\), the median timestep moves from 0.5 to 0.75, where the SNR is \(1/9\).
How to read Figure 5
Start with logit-normal \(m = 0\), \(s = 1\), \(\alpha = 1\): the density peaks at \(t = 0.5\) and 72.8% of the mass is in the band. Raise \(\alpha\) to 3 (SD3 at 1024²): the median moves to 0.75, only 48.6% stays in the band, and 13.6% of steps are now above \(t = 0.9\). Switch to uniform and move \(\alpha\): the shift alone also pushes mass toward noise. A smaller \(s\) concentrates training on fewer noise levels; the model then gets worse at the levels it rarely sees.
The network: MM-DiT
The backbone is a diffusion transformer (session 228) working on patches of a 16-channel VAE latent [Certain]. The new part is how text enters. In a standard DiT with cross-attention, image tokens query text tokens, but text tokens never change. SD3’s MM-DiT (multimodal DiT) gives text tokens and image tokens separate weights (separate norms, QKV projections and MLPs) but runs one joint attention over the concatenated sequence, so information flows both ways. The paper describes it as “equivalent to having two independent transformers for each modality, but joining the sequences of the two modalities for the attention operation” [Certain] (SD3 paper).
Other details from the paper [Certain]: three text encoders (CLIP-L/14, OpenCLIP bigG/14 and T5-XXL); models up to depth 38 and 8B parameters; RMSNorm on Q and K before attention to stop attention logits from growing during high-resolution training in bf16; and validation loss that falls smoothly with model size and tracks image quality. FLUX.1 also calls itself a rectified flow transformer with 12B parameters; its fast “schnell” version generates in 1 to 4 steps after “latent adversarial diffusion distillation” [Certain] (model card).
Flow matching in PyTorch
A complete, deterministic run on the 8-blob data: train 1-rectified flow with logit-normal timesteps, sample with Euler, then reflow once and train 2-rectified flow. It runs in about 15 seconds on a laptop CPU. The printed numbers are from a real run (PyTorch on CPU).
import math, torch torch.manual_seed(0) # toy data: 8 Gaussian blobs on a circle of radius 2, std 0.15 K, R, S = 8, 2.0, 0.15 ang = 2 * math.pi * torch.arange(K) / K CENTERS = torch.stack([R * torch.cos(ang), R * torch.sin(ang)], 1) def sample_data(n): k = torch.randint(0, K, (n,)) return CENTERS[k] + S * torch.randn(n, 2) class VelocityNet(torch.nn.Module): # v_theta(x_t, t): 2-D point + time -> 2-D velocity def __init__(self, h=256): super().__init__() self.net = torch.nn.Sequential( torch.nn.Linear(3, h), torch.nn.SiLU(), torch.nn.Linear(h, h), torch.nn.SiLU(), torch.nn.Linear(h, h), torch.nn.SiLU(), torch.nn.Linear(h, 2)) def forward(self, x, t): return self.net(torch.cat([x, t[:, None]], 1)) def cfm_loss(model, x, eps, t): # formula (229.8) xt = (1 - t[:, None]) * x + t[:, None] * eps # straight path: data at t=0, noise at t=1 target = eps - x # its constant velocity, formula (229.4) return ((model(xt, t) - target) ** 2).mean() def train(model, get_pair, steps=3000, bs=512, logit_normal=True): opt = torch.optim.AdamW(model.parameters(), lr=1e-3) for it in range(steps): x, eps = get_pair(bs) t = torch.sigmoid(torch.randn(bs)) if logit_normal else torch.rand(bs) # (229.16), m=0, s=1 loss = cfm_loss(model, x, eps, t) opt.zero_grad(); loss.backward(); opt.step() return loss.item() @torch.no_grad() def euler_sample(model, eps, n_steps): # Algorithm 2 x, ts = eps.clone(), torch.linspace(1, 0, n_steps + 1) for i in range(n_steps): t = torch.full((x.shape[0],), ts[i].item()) x = x + (ts[i + 1] - ts[i]) * model(x, t) # dt is negative return x def near_modes(x): # share of samples within 3 std of a blob center return (torch.cdist(x, CENTERS).min(1).values < 3 * S).float().mean().item() # 1) 1-rectified flow: independent pairs (data, noise) rf1 = VelocityNet() print("1-RF final loss", round(train(rf1, lambda n: (sample_data(n), torch.randn(n, 2))), 3)) z = torch.randn(2000, 2) for n in [1, 2, 4, 8, 32]: print(f"1-RF steps={n:2d} near modes={near_modes(euler_sample(rf1, z, n)):.3f}") # 2) reflow, Algorithm 3: new pairs (Phi(eps), eps) from the 1-RF ODE, then train again eps_bank = torch.randn(20000, 2) x_bank = euler_sample(rf1, eps_bank, 100) # the deterministic coupling def reflow_pairs(n): i = torch.randint(0, eps_bank.shape[0], (n,)) return x_bank[i], eps_bank[i] rf2 = VelocityNet() print("2-RF final loss", round(train(rf2, reflow_pairs), 3)) for n in [1, 2, 4, 8, 32]: print(f"2-RF steps={n:2d} near modes={near_modes(euler_sample(rf2, z, n)):.3f}") # Output of one run: # 1-RF final loss 1.934 <- the floor C of (229.9), not a bad model # 1-RF steps= 1 near modes=0.000 <- one step = the data mean # 1-RF steps= 2 0.148 | 4 0.702 | 8 0.853 | 32 0.929 # 2-RF final loss 0.001 <- reflow pairs barely cross: straight flow # 2-RF steps= 1 near modes=0.793 # 2-RF steps= 2 0.828 | 4 0.863 | 8 0.876 | 32 0.892 <- better with few steps, capped later
The run shows the three lessons of this session at once. The 1-RF loss stops at 1.93 and the model is still good (93% at 32 steps). One step without reflow gives 0%. After reflow the loss is almost zero, one step gives 79%, but at 32 steps the 2-RF model (89%) is below the 1-RF model, because it learned from the 1-RF model’s 100-step samples and inherits their errors. To use this on images, replace the MLP with a DiT on VAE latents, add the text condition, and use the shift (229.18) for both training and sampling.
Where this is used
| System or task | How flow matching is used | Notes |
|---|---|---|
| Stable Diffusion 3 / 3.5 (text-to-image) | Rectified flow, logit-normal timesteps, shift \(\alpha = 3\) at 1024², MM-DiT up to 8B | The reference recipe for this session [Certain] |
| FLUX.1 (text-to-image) | 12B rectified flow transformer; “schnell” distilled to 1 to 4 steps | Details from the model card and released code [Likely] |
| Video generation (Movie Gen and other video DiTs) | Flow matching on spatio-temporal latent patches; same loss, much longer token sequences | Movie Gen’s 30B model uses flow matching [Likely]; see session 231 |
| Few-step and one-step generators | Reflow (InstaFlow) or distillation of a flow teacher | Straighter paths make distillation easier; see session 230 |
| Robot action generation in vision-language-action models | A small “action expert” turns noise into a chunk of future actions with flow matching (for example \(\pi_0\)) | [Likely]; see session 220 |
| Speech and audio generation | Flow matching on spectrogram or audio latents (for example Voicebox) | [Likely] |
| Synthetic data for video understanding | Generated clips or frames to augment rare classes in action recognition or key-moment detection | A use of the generator, not of the training method; quality checks are needed [Speculation] |
Common mistakes
- Mixing time conventions. Rectified flow (Liu et al.) and Lipman et al. put noise at \(t = 0\) and data at \(t = 1\); SD3 and many libraries put data at \(t = 0\). The velocity changes sign between them. Check which end is noise before you reuse a sampler or a pretrained model.
- Stepping the wrong way. With data at \(t = 0\), sampling goes from \(t = 1\) to 0 and the Euler update is \(\mathbf{x} \leftarrow \mathbf{x} + (t_{i+1} - t_i)\mathbf{v}\) with a negative step. A sign error here produces samples that fly off to infinity.
- Reading the training loss as quality. The CFM loss has a floor \(C\) (229.9) that depends on the data and on how much the pair lines cross. A loss of 1.9 can be a good model; compare models with sample metrics (FID, CLIP score, human preference).
- Dividing by \(t\) or \(1 - t\) at the ends. Converting \(\hat{\mathbf{v}}\) to \(\hat{\epsilon}\) or a score near \(t = 0\) or \(t = 1\) can divide by almost zero (formula 229.12). Clip \(t\) to \([\delta, 1 - \delta]\) or keep the conversion in the stable direction (229.11).
- Different shift in training and sampling. The shift (229.18) changes which noise levels the model learned. SD3 uses the same \(\alpha\) in both. A sampler with a different time grid asks the model about noise levels it rarely saw.
- Expecting one step to work without reflow or distillation. One Euler step of 1-rectified flow returns the data mean (Figure 4). “Rectified flow” in a paper title does not mean the model samples in one step.
- Reflowing with a weak teacher or few pairs. The new model can only be as good as the pairs; errors of the teacher’s samples become the targets (Figure 4, many steps).
- Uniform timesteps at high resolution. Too little training at high noise gives images whose global layout is unstable; use logit-normal sampling with a resolution shift.
Interview questions
What does a flow matching model predict, and how do you generate a sample?basic
Derive the training target of rectified flow.basic
The true velocity at a point is an average over all data. Why is it fine to regress on the velocity of one pair?core
If every training path is a straight line, why does the trained model need many steps?core
What does a single Euler step of a flow matching model produce, and why?core
Is flow matching different from diffusion?core
What is reflow, what does it guarantee, and what does it cost?advanced
Why did SD3 sample training timesteps from a logit-normal distribution?core
Why shift timesteps toward noise at high resolution?advanced
How does MM-DiT differ from a DiT with cross-attention?core
Your flow model gives good images at 50 steps; the product needs 4. What are your options?advanced
Exercises
- Data \(x = 1\), noise \(\epsilon = 0.5\), \(t = 0.4\). Compute \(x_t\) and the rectified flow target. Then recover \(x\) from \(x_t\) and the true velocity.
- Show that one Euler step from \(t = 1\) with the exact marginal velocity returns \(\E[\mathbf{x}]\) for every noise sample. Use \(\mathbf{u}_t(\mathbf{z}) = (\mathbf{z} - \hat{\mathbf{x}}(\mathbf{z}, t))/t\).
- With shift \(\alpha = 3\), where does \(t = 0.5\) go? What are the SNR before and after?
- At \(t = 0.5\), how much larger is the velocity loss than the \(\epsilon\) loss for the same model? At \(t = 0.9\)?
- A sampler uses 28 steps with classifier-free guidance. How many network evaluations does one image cost? How many with 4 steps and guidance distilled into the model?
- In Lipman’s formula (229.13), set \(\sigma_\text{min} = 0\) and replace \(t\) by \(1 - t\). Show that you get formula (229.3), and that Lipman’s velocity \(\mathbf{x}_1 - \mathbf{x}_0\) is minus our velocity \(\epsilon - \mathbf{x}\).
Answerscheck
Key takeaways
- A flow model is an ODE (229.1): a network gives the velocity, and Euler steps (229.2) carry noise at \(t = 1\) to data at \(t = 0\).
- Rectified flow trains on straight lines \(\mathbf{x}_t = (1 - t)\mathbf{x} + t\epsilon\) (229.3) with target \(\epsilon - \mathbf{x}\) (229.4, 229.8). Regressing on single pairs gives the same gradient as matching the averaged velocity (229.6, 229.9), but the loss has a floor.
- Velocity, noise and data predictions are interchangeable (229.11); their losses differ only by time weights (229.12). Flow matching with Gaussian paths and diffusion are one family (229.10).
- The learned flow curves because training lines cross. One step gives the data mean. Reflow (229.14) re-pairs noise with its own output, which straightens the flow and makes 1 to 2 steps work, at some cost in quality at many steps.
- SD3: rectified flow, logit-normal timesteps (229.16) that focus on mid noise (a normal on log-SNR, 229.17), a resolution shift (229.18, \(\alpha = 3\) at 1024²), and the two-stream MM-DiT. FLUX.1 is a 12B rectified flow transformer.
Key formulas
References for this session
- Lipman et al. (2022). Flow Matching for Generative Modeling
- Liu et al. (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- Lipman et al. (2022). Flow Matching for Generative Modeling
- Liu et al. (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- Albergo & Vanden-Eijnden (2022). Building Normalizing Flows with Stochastic Interpolants
- Liu, Zhang, Ma, Peng & Liu (2023). InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation
- Black Forest Labs (2024). FLUX.1 [schnell] model card
- Polyak et al. (Meta) (2024). Movie Gen: A Cast of Media Foundation Models
- Ho et al. (2020). Denoising Diffusion Probabilistic Models
- Song et al. (2020). Denoising Diffusion Implicit Models
- Song et al. (2020). Score-Based Generative Modeling through Stochastic Differential Equations
- Peebles & Xie (2022). Scalable Diffusion Models with Transformers
- Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models
- Ho & Salimans (2022). Classifier-Free Diffusion Guidance
- Song et al. (2023). Consistency Models