LLM & VLM courseBlog
229Track 9: Generative models · session 8 of 12

Flow matching and rectified flow

Flow matching trains a network to predict a velocity: the direction and speed that move a noise sample toward data. Training is a simple regression on straight lines between noise and data, and sampling is solving an ordinary differential equation. Rectified flow makes the paths straighter so fewer steps are needed, and this recipe, with a better choice of training timesteps, is what Stable Diffusion 3 and FLUX use.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Diffusion 1 and 2 (sessions 225 and 226), latent diffusion and DiT (session 228), backpropagation and AdamW (session 4); basic PyTorch

What you will learn

  • What a velocity field and an ODE are, and how an Euler step moves a sample from noise to data
  • The straight path x_t = (1 − t) x + t ε, its constant velocity ε − x, and why the training target is that simple
  • Why the flow matching loss on single pairs has the same gradient as the loss on the true (averaged) velocity, and what the network really learns
  • How flow matching relates to diffusion: x-, ε- and v-prediction are one model with different loss weights
  • Rectified flow and reflow: why learned paths curve, how re-pairing noise with its own output straightens them, and the cost in quality
  • The SD3 recipe: logit-normal timestep sampling, the resolution shift, and the MM-DiT transformer; FLUX.1 as a 12B rectified flow transformer
  • Common mistakes, from the flipped time convention between papers to one-step sampling without reflow

Why this matters

Diffusion models generate by removing noise in many small steps. Flow matching keeps the same idea (start from noise, end at data) but asks the network a simpler question: “at this point and this time, in which direction and how fast should the sample move?” The answer is a velocity. Training is plain regression, and sampling is following the velocity like a boat following a river current.

This is not a side topic. Stable Diffusion 3 (Esser et al., 2024) is trained with rectified flow, the simplest kind of flow matching, and its paper reports that rectified flow with a better choice of training timesteps beat the established diffusion formulations in a large comparison [Certain] (SD3 paper). FLUX.1 is described by its authors as a “12 billion parameter rectified flow transformer” [Certain] (model card). Meta’s Movie Gen video model, a 30B-parameter transformer, is also trained with flow matching [Likely]. If you read about a modern image or video generator, you will meet the words “velocity”, “rectified flow” and “logit-normal”.

The session builds four ideas, each a common interview question:

  • A velocity field moves noise to data. A network \(\mathbf{v}_\theta(\mathbf{x}, t)\) defines an ordinary differential equation (ODE); solving it from \(t = 1\) (noise) to \(t = 0\) (data) generates a sample (formula 229.1).
  • The training loss is simple regression. Pick a data point, a noise sample and a time, mix them on a straight line, and ask the network for the line’s velocity \(\epsilon - \mathbf{x}\) (formula 229.8).
  • Straight paths are cheap to follow. The learned paths are not straight, but reflow makes them straighter, so a few steps are enough (Figures 2 to 4).
  • The SD3 recipe: sample training timesteps from a logit-normal distribution, shift them for high resolution, and use a two-stream transformer (MM-DiT).

We build on session 225 (forward noising, \(\epsilon\)-prediction), session 226 (the probability-flow ODE and DDIM, which are already deterministic flows) and session 228 (the latent space and the DiT backbone). The next session, session 230, pushes the step count down to one or two.

The idea: a velocity field that moves noise to data

Think of every point \(\mathbf{x}\) in space (an image, or a latent) as having an arrow attached: a velocity \(\mathbf{v}(\mathbf{x}, t)\) that says how the point should move at time \(t\). A collection of such arrows for all points is a velocity field. If you drop a particle somewhere and let it follow the arrows, it traces a path. Writing this as an equation gives an ordinary differential equation (ODE):

\[ \frac{d\mathbf{x}_t}{dt} = \mathbf{v}_\theta(\mathbf{x}_t, t), \qquad \mathbf{x}_1 \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) \tag{229.1} \]
Read the symbols
\(t\)
time, from 1 (pure noise) down to 0 (data). We use the convention of SD3 and of the diffusion sessions: \(t = 0\) is data.
\(\mathbf{x}_t\)
the sample at time \(t\): a vector, for example a flattened latent image
\(\mathbf{v}_\theta\)
the network: input a point and a time, output a vector of the same size (the velocity)
\(\mathcal{N}(\mathbf{0}, \mathbf{I})\)
a standard Gaussian: where every sample starts

A model of this kind is called a continuous normalizing flow (CNF). CNFs existed before 2022, but they were trained by simulating the ODE during training, which is slow. Flow matching (Lipman et al., 2022) trains them without simulation: no ODE is solved during training, just one network call per example, like diffusion.

To generate, we solve the ODE backward from \(t = 1\) to \(t = 0\). The simplest solver is Euler’s method: take a small time step \(\Delta\) and move along the current arrow:

\[ \mathbf{x}_{t - \Delta} = \mathbf{x}_t - \Delta \, \mathbf{v}_\theta(\mathbf{x}_t, t) \tag{229.2} \]

The minus sign is there because time goes down. Each step costs one network call, so the number of steps is the cost of generation. Euler is exact when the path is a straight line traveled at constant speed: then one step of size 1 lands exactly on the end. That single fact explains most of this session.

Different from diffusion sampling with DDPM (session 225), there is no fresh noise added during sampling: the same starting noise always gives the same output. In session 226 you met the probability-flow ODE and DDIM, which are also deterministic. Flow matching starts from that ODE view and makes it the whole model.

The path: a straight line between noise and data

To train a velocity field, we first need to say which path a sample should take. Flow matching builds the answer from the simplest possible piece. Take one data point \(\mathbf{x}\) and one noise sample \(\epsilon\), and connect them with a straight line:

\[ \mathbf{x}_t = (1 - t)\, \mathbf{x} + t\, \epsilon, \qquad t \in [0, 1] \tag{229.3} \]

At \(t = 0\) this is the data point, at \(t = 1\) it is the noise. In between it is a mix, exactly like the noisy image \(\sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon\) of DDPM, only with different weights: here the two weights add up to 1 (and the variance dips in the middle), while in DDPM their squares add up to 1. The velocity along this line is its time derivative, and it does not depend on \(t\):

\[ \frac{d\mathbf{x}_t}{dt} = \epsilon - \mathbf{x} \tag{229.4} \]

This is the conditional velocity: the velocity of the path for one given pair \((\mathbf{x}, \epsilon)\). It is constant, so this path is straight and traveled at constant speed. This kind of path is what Liu et al. (2022) call rectified flow and what Lipman et al. (2022) call the optimal-transport (OT) conditional path. Albergo and Vanden-Eijnden (2022) proposed the same kind of construction under the name stochastic interpolants. Three groups, the same year, the same idea.

Worked example: one training target

One number, so you can check by hand. Data \(x = 2\), noise \(\epsilon = -1\), time \(t = 0.25\).

The point on the line: \(x_t = 0.75 \times 2 + 0.25 \times (-1) = 1.25\). The target velocity: \(\epsilon - x = -1 - 2 = -3\). If the network outputs \(\hat v = -2.5\), the squared error is \((-2.5 - (-3))^2 = 0.25\).

From a velocity you can read off both endpoints: \(\hat x = x_t - t \hat v\) and \(\hat\epsilon = x_t + (1 - t)\hat v\). With the true \(v = -3\): \(\hat x = 1.25 + 0.75 = 2.0\) and \(\hat\epsilon = 1.25 - 2.25 = -1.0\). Both are exact, as they should be (formula 229.11 below).

Many pairs give many lines, and together they move a whole distribution. At each time \(t\), the points \(\mathbf{x}_t\) of all pairs form a distribution \(p_t\): the data at \(t = 0\), the Gaussian at \(t = 1\), and a blend in between. This family \(p_t\) is called the probability path. A velocity field \(\mathbf{u}_t\) moves the distribution along the path if mass is conserved, which is the continuity equation from physics:

\[ \frac{\partial p_t(\mathbf{z})}{\partial t} + \nabla \cdot \big( p_t(\mathbf{z})\, \mathbf{u}_t(\mathbf{z}) \big) = 0 \tag{229.5} \]

In words: the density at a point changes only because probability flows in or out. You do not need to compute with this equation; it is the guarantee behind the method. If the network learns the right \(\mathbf{u}_t\), the ODE (229.1) carries Gaussian noise exactly onto the data distribution.

The flow matching loss: why simple regression works

Here is the problem. Many straight lines pass through the same point \(\mathbf{z}\) at the same time \(t\): different pairs \((\mathbf{x}, \epsilon)\) can produce the same \(\mathbf{x}_t\). Each of these lines has a different velocity. But an ODE needs one velocity per point. The right one is the average of the velocities of all lines through that point, weighted by how likely each line is:

\[ \mathbf{u}_t(\mathbf{z}) = \E\big[\, \epsilon - \mathbf{x} \;\big|\; \mathbf{x}_t = \mathbf{z} \,\big] \tag{229.6} \]

This is the marginal velocity. It is the field that satisfies the continuity equation (229.5) for the path \(p_t\) [Certain] (Lipman et al., Theorem 1; Liu et al.). The natural loss would be to match it directly, the flow matching (FM) loss:

\[ \mathcal{L}_\text{FM}(\theta) = \E_{t,\; \mathbf{z} \sim p_t} \big\lVert \mathbf{v}_\theta(\mathbf{z}, t) - \mathbf{u}_t(\mathbf{z}) \big\rVert^2 \tag{229.7} \]

We cannot compute \(\mathbf{u}_t(\mathbf{z})\): it is an average over the whole dataset. The trick is to regress on the velocity of one line instead, the conditional flow matching (CFM) loss:

\[ \mathcal{L}_\text{CFM}(\theta) = \E_{t,\; \mathbf{x} \sim p_\text{data},\; \epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})} \big\lVert \mathbf{v}_\theta\big((1 - t)\mathbf{x} + t \epsilon,\; t\big) - (\epsilon - \mathbf{x}) \big\rVert^2 \tag{229.8} \]
Read the symbols
\(t\)
a random time in \([0, 1]\); uniform in the basic method, logit-normal in SD3 (formula 229.16)
\(\mathbf{x} \sim p_\text{data}\)
a random training example (in latent diffusion, a VAE latent)
\(\epsilon\)
fresh Gaussian noise of the same shape
\(\epsilon - \mathbf{x}\)
the target: the velocity of the straight line through this pair

The key result: the two losses differ by a constant that does not depend on \(\theta\), so they have the same gradient:

\[ \mathcal{L}_\text{FM}(\theta) = \mathcal{L}_\text{CFM}(\theta) - C, \qquad \nabla_\theta \mathcal{L}_\text{FM} = \nabla_\theta \mathcal{L}_\text{CFM} \tag{229.9} \]

Why? This is the same fact you use whenever you train with mean squared error. For a fixed input, the prediction that minimizes \(\E\lVert \mathbf{v} - \mathbf{y}\rVert^2\) over noisy targets \(\mathbf{y}\) is their conditional mean \(\E[\mathbf{y} \mid \text{input}]\). Expand the square: \(\lVert \mathbf{v} - \mathbf{y}\rVert^2 = \lVert \mathbf{v} - \E[\mathbf{y}]\rVert^2 + \lVert \E[\mathbf{y}] - \mathbf{y}\rVert^2 + \) a cross term whose average is zero. The second term is the constant \(C\): the spread of the targets of all lines through the same point. So a network trained on noisy single-pair targets converges to the average velocity (229.6) [Certain] (Lipman et al., Theorem 2).

Worked example: the network learns an average of crossing targets

Data is \(+1\) or \(-1\) with equal probability; noise is \(\mathcal{N}(0, 1)\). Look at the point \(z = 0.2\) at time \(t = 0.5\). Two lines pass through it:

From \(x = +1\): the noise must be \(\epsilon = (z - (1 - t)x)/t = (0.2 - 0.5)/0.5 = -0.6\), so the target is \(\epsilon - x = -1.6\). From \(x = -1\): \(\epsilon = (0.2 + 0.5)/0.5 = 1.4\), target \(1.4 + 1 = 2.4\).

How likely is each line? Given \(x\), the point \(x_t\) is Gaussian with mean \((1 - t)x\) and standard deviation \(t\). This gives weights \(0.690\) for \(x = +1\) and \(0.310\) for \(x = -1\). The marginal velocity is the weighted average: \(0.690 \times (-1.6) + 0.310 \times 2.4 = -0.360\).

For two-point data there is a closed form: the best guess of the data is \(\hat x = \tanh\big((1 - t) z / t^2\big) = \tanh(0.4) = 0.380\), and \(u = (z - \hat x)/t = (0.2 - 0.380)/0.5 = -0.360\). Same number. The network is trained on targets \(-1.6\) and \(2.4\) but learns \(-0.360\), a velocity that no single training line has.

The last line of that example is the general formula for our straight path. Since \(\epsilon = (\mathbf{x}_t - (1 - t)\mathbf{x})/t\), the marginal velocity can be written as \(\mathbf{u}_t(\mathbf{z}) = (\mathbf{z} - \hat{\mathbf{x}}(\mathbf{z}, t))/t\), where \(\hat{\mathbf{x}} = \E[\mathbf{x} \mid \mathbf{x}_t = \mathbf{z}]\) is the best guess of the clean data. For data that is a mixture of Gaussians, \(\hat{\mathbf{x}}\) has a closed form, so we can compute the exact optimal velocity field. Figures 1 to 4 do this for 2-D data made of 8 Gaussian blobs on a circle (radius 2, standard deviation 0.15). No trained network is involved in Figure 1: it shows what a perfectly trained network would learn.

Show
Figure 1. Forty noise points (small circles). Orange: the straight training lines to randomly paired data points. Dark lines (light in dark mode): the paths of the exact marginal velocity field (229.6) from the same noise points, solved with 200 Euler steps. Thin gray arrows: the direction a sample moves next at the chosen time.
How to read Figure 1

Press Play, or move the time slider from 1 to 0 yourself. The orange dots slide along straight lines, but the lines cross everywhere: at a crossing, two training targets disagree, and the network can only learn their average. The dark dots follow that average. Their paths never cross, because an ODE has one velocity per point. Notice that many of these paths first move inward, toward the center of the data, and only later turn toward one blob. Early on (\(t\) near 1) the noisy point says almost nothing about which blob it will reach, so the best guess \(\hat{\mathbf{x}}\) is near the data mean. As \(t\) falls, the guess commits to one blob and the path bends. The training paths are straight; the learned paths are not.

Training is now easy to write down. It looks almost exactly like DDPM training from session 225: only the mixing weights and the target change.

Algorithm 1: flow matching (rectified flow) training
  1. input: dataset (images or VAE latents), network \(\mathbf{v}_\theta\), timestep distribution \(\pi(t)\)
  2. repeat
  3.   sample a batch \(\mathbf{x}\) from the data, \(\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) of the same shape, \(t \sim \pi(t)\) // uniform, or logit-normal as in SD3
  4.   \(\mathbf{x}_t \leftarrow (1 - t)\,\mathbf{x} + t\,\epsilon\) // formula (229.3)
  5.   \(L \leftarrow \lVert \mathbf{v}_\theta(\mathbf{x}_t, t) - (\epsilon - \mathbf{x}) \rVert^2\), averaged over the batch // formula (229.8)
  6.   take an AdamW step on \(\nabla_\theta L\)
  7. until samples are good (not until the loss is near zero: it cannot be)

Sampling: solve the ODE

Sampling is Euler’s method (229.2), run from \(t = 1\) to \(t = 0\):

Algorithm 2: Euler sampling with \(N\) steps
  1. input: trained \(\mathbf{v}_\theta\), number of steps \(N\), optional time grid \(1 = t_0 \gt t_1 \gt \dots \gt t_N = 0\) // uniform, or shifted (229.18)
  2. \(\mathbf{x} \leftarrow \epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\)
  3. for \(i = 0, \dots, N - 1\):
  4.   \(\mathbf{x} \leftarrow \mathbf{x} + (t_{i+1} - t_i)\, \mathbf{v}_\theta(\mathbf{x}, t_i)\) // \(t_{i+1} - t_i \lt 0\): we move against the velocity
  5. return \(\mathbf{x}\) // a latent: decode it with the VAE in latent models

With classifier-free guidance (session 227), each step calls the network twice, with and without the prompt, and uses \(\mathbf{v}_\text{uncond} + w(\mathbf{v}_\text{cond} - \mathbf{v}_\text{uncond})\). Higher-order solvers such as Heun (two network calls per step) reduce the error per step; adaptive solvers such as Dormand–Prince choose the step size themselves. Lipman et al. stress that off-the-shelf ODE solvers work well for these models [Certain] (abstract).

Figure 2 runs Algorithm 2 in your browser on the same 8-blob data, one line at a time. The velocity is the exact marginal velocity (229.6) of 1-rectified flow, or the 2-rectified flow that reflow produces (built in Figure 3). Every number in the panel is computed live: the velocity at the tracked sample, the guess \(\hat{\mathbf{x}}\) of formula (229.11), and the update (229.2) with the real values put in.

Model
samplestracked: Euler stepstrue path (many steps)guess \(\hat{\mathbf{x}}\)
  1. input: \(\mathbf{v}_\theta\), \(N\); grid \(t_i = 1 - i/N\)
  2. \(\mathbf{x} \leftarrow \epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\)
  3. for \(i = 0, \dots, N - 1\):
  4.   \(\mathbf{v} \leftarrow \mathbf{v}_\theta(\mathbf{x}, t_i)\) // one network call
  5.   \(\mathbf{x} \leftarrow \mathbf{x} + (t_{i+1} - t_i)\,\mathbf{v}\) // (229.2)
  6. return \(\mathbf{x}\)
Figure 2. Euler sampling, one line of Algorithm 2 per step. Blue dots: 200 noise samples. Gray arrows: the direction \(-\mathbf{v}\) each point would move at the current time. Orange: the tracked sample and its straight Euler segments; thin orange lines show where every sample goes in the coming step. Dashed purple: the true path of the tracked sample (200 Euler steps for 1-RF, 100 for 2-RF). Click or tap any dot to track it instead.
How to read Figure 2

Press Play with \(N = 4\) and 1-rectified flow. The tracked sample starts at \(\epsilon = (-0.190, 0.475)\). In the first step, at \(t = 1\), the guess \(\hat{\mathbf{x}}\) is the data mean \((0, 0)\), so \(\mathbf{v} = \epsilon\) and the sample moves a quarter of the way to the center, to \((-0.142, 0.356)\). At \(t = 0.5\) the guess has moved toward the top blob, and in the last step the velocity is large, \((1.297, -3.560)\): the sample jumps to \((-0.606, 1.597)\). The true path ends at \((-0.275, 1.765)\), so the four straight steps miss by 0.372; 79% of all samples land in a blob. With \(N = 16\) the miss is 0.055 and 99% land in a blob. With \(N = 1\) every sample goes to the center: 0%. Now switch to the 2-rectified flow: one step already puts 70% of the samples into blobs, and two steps 81%, because its paths are almost straight.

Why one step gives a blurry average

What does a single Euler step from \(t = 1\) produce? It is \(\mathbf{x} = \epsilon - \mathbf{u}_1(\epsilon)\). At \(t = 1\) the noisy point carries no information about the data, so \(\hat{\mathbf{x}}(\epsilon, 1) = \E[\mathbf{x}]\), the data mean, and \(\mathbf{u}_1(\epsilon) = \epsilon - \E[\mathbf{x}]\). So one step sends every noise sample to the data mean. For images, that is a gray blur. This is the price of the averaging in (229.6): the learned paths curve, and a single straight step cannot follow a curve. Figure 4 shows it: 0% of one-step samples land in a data blob.

Flow matching and diffusion: one family

Is flow matching a new kind of model, or diffusion in different clothes? Both. Lipman et al. define the method for a general family of Gaussian paths, and diffusion paths are members [Certain] (abstract):

\[ \mathbf{x}_t = a_t \mathbf{x} + b_t \epsilon, \qquad \frac{d\mathbf{x}_t}{dt} = a'_t \mathbf{x} + b'_t \epsilon \tag{229.10} \]

Rectified flow is \(a_t = 1 - t\), \(b_t = t\). The variance-preserving diffusion path of DDPM has \(a_t^2 + b_t^2 = 1\), for example \(a_t = \cos(\pi t/2)\), \(b_t = \sin(\pi t/2)\) (the cosine schedule in continuous time). Training any of them with velocity regression gives a deterministic ODE that is the probability-flow ODE of session 226 for that path. What changes between “diffusion” and “flow matching” is (1) the path, (2) what the network outputs, and (3) how the loss weights the different times.

Point (2) is not a real difference, because the three outputs convert into each other. For the straight path, given \(\mathbf{x}_t\) and a predicted velocity \(\hat{\mathbf{v}}\):

\[ \hat{\mathbf{x}} = \mathbf{x}_t - t\, \hat{\mathbf{v}}, \qquad \hat{\epsilon} = \mathbf{x}_t + (1 - t)\, \hat{\mathbf{v}} \tag{229.11} \]

Put the true values in and subtract: the errors are tied together by fixed factors, so the three losses are the same loss with different weights over time:

\[ \lVert \mathbf{v} - \hat{\mathbf{v}} \rVert^2 = \frac{\lVert \epsilon - \hat{\epsilon} \rVert^2}{(1 - t)^2} = \frac{\lVert \mathbf{x} - \hat{\mathbf{x}} \rVert^2}{t^2} \tag{229.12} \]

At \(t = 0.5\) the velocity loss is 4 times the \(\epsilon\) loss and 4 times the \(\mathbf{x}\) loss. Near \(t = 1\) (almost pure noise), the velocity loss puts a huge weight on the \(\epsilon\) error; near \(t = 0\), on the \(\mathbf{x}\) error. The SD3 paper compares its formulations in this way, as one loss with different time weightings [Likely].

For completeness, Lipman et al. use the opposite time convention (\(t = 0\) noise, \(t = 1\) data) and add a tiny noise level \(\sigma_\text{min}\) at the data end. In their notation, with \(\mathbf{x}_0\) the noise and \(\mathbf{x}_1\) the data:

\[ \psi_t(\mathbf{x}_0) = \big(1 - (1 - \sigma_\text{min})\,t\big)\,\mathbf{x}_0 + t\,\mathbf{x}_1, \qquad \frac{d\psi_t}{dt} = \mathbf{x}_1 - (1 - \sigma_\text{min})\,\mathbf{x}_0 \tag{229.13} \]

With \(\sigma_\text{min} = 0\) and the time flipped (\(t \to 1 - t\)), this is our formula (229.3), and the velocity is ours with the sign flipped. Keep this in mind when you read code: the sign of the velocity depends on the convention.

Rectified flow: making the paths straight

Figure 1 showed the problem: the training lines are straight, but they cross, so the learned flow curves, and curves need many steps. Liu, Gong and Liu (2022) asked: can we pick the pairs so that the lines do not cross? If no two lines cross, there is no averaging, the learned velocity is the velocity of the one line through each point, and the flow follows straight lines. Then one Euler step is exact.

The pairing of noise with data is called a coupling. Basic flow matching uses the independent coupling: any noise with any data. Rectified flow’s answer is reflow: use the trained flow itself to create a better coupling. Run the ODE from each noise sample \(\epsilon\) to its output \(\Phi_k(\epsilon)\), and train a new flow on the pairs \((\Phi_k(\epsilon), \epsilon)\):

\[ \mathbf{v}^{(k+1)} = \argmin_{\mathbf{v}} \; \E_{t,\, \epsilon} \big\lVert \mathbf{v}\big((1 - t)\Phi_k(\epsilon) + t\epsilon,\; t\big) - \big(\epsilon - \Phi_k(\epsilon)\big) \big\rVert^2 \tag{229.14} \]
Read the symbols
\(\Phi_k(\epsilon)\)
the sample that the \(k\)-th flow produces from noise \(\epsilon\) (solve its ODE from \(t = 1\) to 0)
\(\mathbf{v}^{(1)}\)
“1-rectified flow”: plain flow matching with independent pairs
\(\mathbf{v}^{(2)}\)
“2-rectified flow”: trained once more on its own pairs

Why does this help? The paths of an ODE never cross. So the new pairs \((\Phi_k(\epsilon), \epsilon)\) are already connected by non-crossing (curved) paths, and the straight lines between them cross much less. Liu et al. prove two properties [Certain] (rectified flow paper): the new coupling still maps the noise distribution exactly onto the same data distribution as the old flow, and the transport cost \(\E\, c(\Phi(\epsilon) - \epsilon)\) does not increase for every convex cost \(c\). They measure how far a flow is from straight with the straightness:

\[ S(\mathbf{v}) = \int_0^1 \E \big\lVert (\epsilon - \Phi(\epsilon)) - \mathbf{v}(\mathbf{z}_t, t) \big\rVert^2 \, dt \tag{229.15} \]

where \(\mathbf{z}_t\) is the path from \(\epsilon\). \(S = 0\) means every path is a straight line at constant speed. Notice that (229.15) on the reflow pairs is just the CFM loss on those pairs: a reflowed model has a much smaller loss floor \(C\). In the PyTorch code below, the loss drops from 1.93 to 0.001 after reflow.

  1. train \(\mathbf{v}^{(1)}\) on random pairs \((\mathbf{x}, \epsilon)\) // Algorithm 1
  2. \(\Phi_1(\epsilon_j) \leftarrow\) solve the ODE of \(\mathbf{v}^{(1)}\) from each \(\epsilon_j\)
  3. new pairs \((\Phi_1(\epsilon_j), \epsilon_j)\), straight lines // (229.14)
  4. train \(\mathbf{v}^{(2)}\) on them; sample from fresh noise
Figure 3. Reflow on the 8-blob data. Steps 1 and 3 count crossings among 60 pair lines and give the mean squared distance between paired points over 1,500 pairs. Steps 2 and 4 show the ODE paths and how much longer they are than the straight segment between their ends. In each step the dots travel from noise to data along the lines or paths of that stage; the panel follows pair #1.
How to read Figure 3

Compare step 1 and step 3: the same noise points, the same target distribution, but a new partner for each noise point. The partners of step 3 are much closer (the mean squared distance drops by about 7 times) and the lines barely cross. In step 4 the new flow is trained on those lines. The 1-RF velocity is exact here; the 2-RF velocity is estimated by kernel regression on the 1,500 reflow pairs (each query averages the targets of nearby pair lines, bandwidth 0.1), which plays the role of a trained network. The 2-RF paths start from fresh noise points that were not in the training pairs.

Straight paths pay off at sampling time. Figure 4 runs Euler’s method with \(N\) steps for the exact 1-rectified flow and for the 2-rectified flow, from the same 400 noise points.

Model
Figure 4. Steps vs quality. Dots: 400 samples after \(N\) Euler steps; lines: 30 of their paths. Right: the share of samples within 3 standard deviations of a blob center, for both models at the same \(N\), and the mean distance to the 400-step result of 1-rectified flow from the same noise.
How to read Figure 4

At \(N = 1\), 1-rectified flow sends every sample to the center (the data mean): 0% in a blob. The 2-rectified flow already puts about three quarters of the samples into blobs with one step. At 3 or 4 steps 1-RF catches up fast, and from about 8 steps it is better: 2-RF stops near 94%, because it can only be as good as the pairs it was trained on and its own fit to them (here the kernel smoothing). This is the honest trade-off of reflow: much better at 1 to 2 steps, a small loss of quality at many steps.

Algorithm 3: reflow (\(k\)-rectified flow to \((k+1)\)-rectified flow)
  1. input: trained flow \(\mathbf{v}^{(k)}\), ODE solver with enough steps for good samples
  2. draw many noise samples \(\epsilon_1, \dots, \epsilon_M\) // for text-to-image, also a prompt for each
  3. \(\mathbf{x}_j \leftarrow \Phi_k(\epsilon_j)\) by solving the ODE of \(\mathbf{v}^{(k)}\) // expensive: \(M\) full generations
  4. train \(\mathbf{v}^{(k+1)}\) with Algorithm 1, but on the fixed pairs \((\mathbf{x}_j, \epsilon_j)\) instead of random pairs // formula (229.14); often initialized from \(\mathbf{v}^{(k)}\)
  5. optional: distill \(\mathbf{v}^{(k+1)}\) into a one-step model by regressing \(\Phi_{k+1}(\epsilon)\) from \(\epsilon\) directly

Reflow is expensive: you must generate a large synthetic dataset with the teacher, and errors of the teacher are inherited (Figure 4). InstaFlow (Liu et al., 2023) applied reflow plus distillation to Stable Diffusion and turned it into a one-step text-to-image model; the authors report that reflow “plays a critical role in improving the assignment between noise and images” [Certain] (InstaFlow abstract). Most large production models, including SD3, do not run reflow; they train 1-rectified flow and then use other distillation methods to reduce steps [Likely]. Those methods are the subject of session 230.

Rectified-flow transformers: the SD3 recipe

Esser et al. (2024) trained rectified flow at scale for text-to-image (SD3), with three changes that matter: which timesteps to train on, how to adapt to resolution, and the network architecture.

Which timesteps to train on: logit-normal sampling

With uniform \(t\), every noise level gets the same number of training examples. But not all levels are equally hard. Near \(t = 0\) the answer is almost given (little noise to remove); near \(t = 1\) the velocity is almost \(\epsilon - \E[\mathbf{x}]\), easy to learn. The SD3 paper notes that predicting the velocity is hardest for \(t\) in the middle of \([0, 1]\), and puts more training weight there with a logit-normal distribution [Certain] (SD3 paper). Sample \(u \sim \mathcal{N}(m, s^2)\) and set \(t = 1/(1 + e^{-u})\). Its density is:

\[ \pi_\text{ln}(t; m, s) = \frac{1}{s\sqrt{2\pi}} \, \frac{1}{t(1 - t)} \, \exp\!\left( -\frac{(\operatorname{logit}(t) - m)^2}{2 s^2} \right), \qquad \operatorname{logit}(t) = \log\frac{t}{1 - t} \tag{229.16} \]

In the SD3 comparison of many formulations and timestep samplers, rectified flow with logit-normal \(m = 0\), \(s = 1\) had the best average rank [Certain] (SD3 paper). With these values, 72.8% of training timesteps fall in the middle band \(0.25 \le t \le 0.75\), against 50% for uniform, and only 1.4% fall above \(t = 0.9\). Figure 5 lets you change \(m\) and \(s\).

There is a neat way to see what logit-normal means. The signal-to-noise ratio of the straight path is the squared ratio of the signal weight to the noise weight:

\[ \text{SNR}(t) = \left( \frac{1 - t}{t} \right)^2, \qquad \lambda = \log \text{SNR}(t) = -2\,\operatorname{logit}(t) \tag{229.17} \]

So a normal distribution on \(\operatorname{logit}(t)\) is a normal distribution on the log-SNR \(\lambda\): logit-normal sampling with \(m = 0\), \(s = 1\) means \(\lambda \sim \mathcal{N}(0, 4)\), centered where signal and noise are equally strong. Choosing a timestep distribution is choosing which noise levels the model practices on, the same question as the noise schedule in session 225.

Higher resolution needs more noise: the timestep shift

Neighboring pixels of a large image are strongly correlated. At the same \(t\), a 1024×1024 image keeps much more recognizable structure than a 256×256 one, because averaging many correlated noisy pixels removes noise. So a high-resolution model needs to spend more of its time at higher noise. SD3 remaps the timesteps with a shift \(\alpha\):

\[ t' = \frac{\alpha\, t}{1 + (\alpha - 1)\, t} \quad\Longleftrightarrow\quad \operatorname{logit}(t') = \operatorname{logit}(t) + \log \alpha \tag{229.18} \]

The paper motivates \(\alpha = \sqrt{m/n}\) for going from \(n\) to \(m\) pixels (that would be 4 from 256² to 1024²), but chose \(\alpha = 3.0\) for 1024×1024 from a human preference study, “both during training and sampling” [Certain] (SD3 paper). The right-hand form shows what the shift does: it adds \(\log\alpha\) to the logit, so it moves a logit-normal distribution from \(m\) to \(m + \log \alpha\), and it divides the SNR at every step by \(\alpha^2\). With \(\alpha = 3\), the median timestep moves from 0.5 to 0.75, where the SNR is \(1/9\).

Base distribution
Figure 5. The density of training timesteps \(t\) (orange) after the shift (229.18). The green band is \(0.25 \le t \le 0.75\); the dashed gray line is the uniform density. All numbers are computed in closed form.
How to read Figure 5

Start with logit-normal \(m = 0\), \(s = 1\), \(\alpha = 1\): the density peaks at \(t = 0.5\) and 72.8% of the mass is in the band. Raise \(\alpha\) to 3 (SD3 at 1024²): the median moves to 0.75, only 48.6% stays in the band, and 13.6% of steps are now above \(t = 0.9\). Switch to uniform and move \(\alpha\): the shift alone also pushes mass toward noise. A smaller \(s\) concentrates training on fewer noise levels; the model then gets worse at the levels it rarely sees.

The network: MM-DiT

The backbone is a diffusion transformer (session 228) working on patches of a 16-channel VAE latent [Certain]. The new part is how text enters. In a standard DiT with cross-attention, image tokens query text tokens, but text tokens never change. SD3’s MM-DiT (multimodal DiT) gives text tokens and image tokens separate weights (separate norms, QKV projections and MLPs) but runs one joint attention over the concatenated sequence, so information flows both ways. The paper describes it as “equivalent to having two independent transformers for each modality, but joining the sequences of the two modalities for the attention operation” [Certain] (SD3 paper).

text tokensT5-XXL + CLIP, per token image tokensnoisy latent patches xt conditioning vectortimestep t + pooled CLIP norm + modulate (text) norm + modulate (image) Q K V (text weights) Q K V (image weights) joint attentionover [text ; image], QK RMSNorm out-proj + MLPtext weights, + residual out-proj + MLPimage weights, + residual split back into two streams
Figure 6. One MM-DiT block, as described in the SD3 paper. Text and image tokens keep their own weights; they meet only inside one attention over the joined sequence. The conditioning vector (timestep embedding plus pooled CLIP text embedding) sets the scale, shift and gate of each stream’s norms (adaLN modulation, dashed). The block outputs a velocity for every image token after the last layer.

Other details from the paper [Certain]: three text encoders (CLIP-L/14, OpenCLIP bigG/14 and T5-XXL); models up to depth 38 and 8B parameters; RMSNorm on Q and K before attention to stop attention logits from growing during high-resolution training in bf16; and validation loss that falls smoothly with model size and tracks image quality. FLUX.1 also calls itself a rectified flow transformer with 12B parameters; its fast “schnell” version generates in 1 to 4 steps after “latent adversarial diffusion distillation” [Certain] (model card).

Flow matching in PyTorch

A complete, deterministic run on the 8-blob data: train 1-rectified flow with logit-normal timesteps, sample with Euler, then reflow once and train 2-rectified flow. It runs in about 15 seconds on a laptop CPU. The printed numbers are from a real run (PyTorch on CPU).

import math, torch
torch.manual_seed(0)

# toy data: 8 Gaussian blobs on a circle of radius 2, std 0.15
K, R, S = 8, 2.0, 0.15
ang = 2 * math.pi * torch.arange(K) / K
CENTERS = torch.stack([R * torch.cos(ang), R * torch.sin(ang)], 1)
def sample_data(n):
    k = torch.randint(0, K, (n,))
    return CENTERS[k] + S * torch.randn(n, 2)

class VelocityNet(torch.nn.Module):          # v_theta(x_t, t): 2-D point + time -> 2-D velocity
    def __init__(self, h=256):
        super().__init__()
        self.net = torch.nn.Sequential(
            torch.nn.Linear(3, h), torch.nn.SiLU(), torch.nn.Linear(h, h), torch.nn.SiLU(),
            torch.nn.Linear(h, h), torch.nn.SiLU(), torch.nn.Linear(h, 2))
    def forward(self, x, t):
        return self.net(torch.cat([x, t[:, None]], 1))

def cfm_loss(model, x, eps, t):                 # formula (229.8)
    xt = (1 - t[:, None]) * x + t[:, None] * eps    # straight path: data at t=0, noise at t=1
    target = eps - x                                 # its constant velocity, formula (229.4)
    return ((model(xt, t) - target) ** 2).mean()

def train(model, get_pair, steps=3000, bs=512, logit_normal=True):
    opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
    for it in range(steps):
        x, eps = get_pair(bs)
        t = torch.sigmoid(torch.randn(bs)) if logit_normal else torch.rand(bs)   # (229.16), m=0, s=1
        loss = cfm_loss(model, x, eps, t)
        opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

@torch.no_grad()
def euler_sample(model, eps, n_steps):          # Algorithm 2
    x, ts = eps.clone(), torch.linspace(1, 0, n_steps + 1)
    for i in range(n_steps):
        t = torch.full((x.shape[0],), ts[i].item())
        x = x + (ts[i + 1] - ts[i]) * model(x, t)  # dt is negative
    return x

def near_modes(x):                                # share of samples within 3 std of a blob center
    return (torch.cdist(x, CENTERS).min(1).values < 3 * S).float().mean().item()

# 1) 1-rectified flow: independent pairs (data, noise)
rf1 = VelocityNet()
print("1-RF final loss", round(train(rf1, lambda n: (sample_data(n), torch.randn(n, 2))), 3))
z = torch.randn(2000, 2)
for n in [1, 2, 4, 8, 32]:
    print(f"1-RF  steps={n:2d}  near modes={near_modes(euler_sample(rf1, z, n)):.3f}")

# 2) reflow, Algorithm 3: new pairs (Phi(eps), eps) from the 1-RF ODE, then train again
eps_bank = torch.randn(20000, 2)
x_bank = euler_sample(rf1, eps_bank, 100)            # the deterministic coupling
def reflow_pairs(n):
    i = torch.randint(0, eps_bank.shape[0], (n,))
    return x_bank[i], eps_bank[i]
rf2 = VelocityNet()
print("2-RF final loss", round(train(rf2, reflow_pairs), 3))
for n in [1, 2, 4, 8, 32]:
    print(f"2-RF  steps={n:2d}  near modes={near_modes(euler_sample(rf2, z, n)):.3f}")

# Output of one run:
# 1-RF final loss 1.934                      <- the floor C of (229.9), not a bad model
# 1-RF steps= 1 near modes=0.000             <- one step = the data mean
# 1-RF steps= 2 0.148 | 4 0.702 | 8 0.853 | 32 0.929
# 2-RF final loss 0.001                      <- reflow pairs barely cross: straight flow
# 2-RF steps= 1 near modes=0.793
# 2-RF steps= 2 0.828 | 4 0.863 | 8 0.876 | 32 0.892   <- better with few steps, capped later

The run shows the three lessons of this session at once. The 1-RF loss stops at 1.93 and the model is still good (93% at 32 steps). One step without reflow gives 0%. After reflow the loss is almost zero, one step gives 79%, but at 32 steps the 2-RF model (89%) is below the 1-RF model, because it learned from the 1-RF model’s 100-step samples and inherits their errors. To use this on images, replace the MLP with a DiT on VAE latents, add the text condition, and use the shift (229.18) for both training and sampling.

Where this is used

System or taskHow flow matching is usedNotes
Stable Diffusion 3 / 3.5 (text-to-image)Rectified flow, logit-normal timesteps, shift \(\alpha = 3\) at 1024², MM-DiT up to 8BThe reference recipe for this session [Certain]
FLUX.1 (text-to-image)12B rectified flow transformer; “schnell” distilled to 1 to 4 stepsDetails from the model card and released code [Likely]
Video generation (Movie Gen and other video DiTs)Flow matching on spatio-temporal latent patches; same loss, much longer token sequencesMovie Gen’s 30B model uses flow matching [Likely]; see session 231
Few-step and one-step generatorsReflow (InstaFlow) or distillation of a flow teacherStraighter paths make distillation easier; see session 230
Robot action generation in vision-language-action modelsA small “action expert” turns noise into a chunk of future actions with flow matching (for example \(\pi_0\))[Likely]; see session 220
Speech and audio generationFlow matching on spectrogram or audio latents (for example Voicebox)[Likely]
Synthetic data for video understandingGenerated clips or frames to augment rare classes in action recognition or key-moment detectionA use of the generator, not of the training method; quality checks are needed [Speculation]

Common mistakes

  • Mixing time conventions. Rectified flow (Liu et al.) and Lipman et al. put noise at \(t = 0\) and data at \(t = 1\); SD3 and many libraries put data at \(t = 0\). The velocity changes sign between them. Check which end is noise before you reuse a sampler or a pretrained model.
  • Stepping the wrong way. With data at \(t = 0\), sampling goes from \(t = 1\) to 0 and the Euler update is \(\mathbf{x} \leftarrow \mathbf{x} + (t_{i+1} - t_i)\mathbf{v}\) with a negative step. A sign error here produces samples that fly off to infinity.
  • Reading the training loss as quality. The CFM loss has a floor \(C\) (229.9) that depends on the data and on how much the pair lines cross. A loss of 1.9 can be a good model; compare models with sample metrics (FID, CLIP score, human preference).
  • Dividing by \(t\) or \(1 - t\) at the ends. Converting \(\hat{\mathbf{v}}\) to \(\hat{\epsilon}\) or a score near \(t = 0\) or \(t = 1\) can divide by almost zero (formula 229.12). Clip \(t\) to \([\delta, 1 - \delta]\) or keep the conversion in the stable direction (229.11).
  • Different shift in training and sampling. The shift (229.18) changes which noise levels the model learned. SD3 uses the same \(\alpha\) in both. A sampler with a different time grid asks the model about noise levels it rarely saw.
  • Expecting one step to work without reflow or distillation. One Euler step of 1-rectified flow returns the data mean (Figure 4). “Rectified flow” in a paper title does not mean the model samples in one step.
  • Reflowing with a weak teacher or few pairs. The new model can only be as good as the pairs; errors of the teacher’s samples become the targets (Figure 4, many steps).
  • Uniform timesteps at high resolution. Too little training at high noise gives images whose global layout is unstable; use logit-normal sampling with a resolution shift.

Interview questions

What does a flow matching model predict, and how do you generate a sample?basic
It predicts a velocity \(\mathbf{v}_\theta(\mathbf{x}_t, t)\): the direction and speed the sample should move at time \(t\). To generate, draw Gaussian noise at \(t = 1\) and solve the ODE \(d\mathbf{x}/dt = \mathbf{v}_\theta\) back to \(t = 0\), usually with Euler or Heun steps. Each step is one network call (two with classifier-free guidance). No noise is added during sampling, so it is deterministic given the starting noise.
Derive the training target of rectified flow.basic
The path is \(\mathbf{x}_t = (1 - t)\mathbf{x} + t\epsilon\). Its derivative in \(t\) is \(\epsilon - \mathbf{x}\), constant in time. So the loss is \(\lVert \mathbf{v}_\theta(\mathbf{x}_t, t) - (\epsilon - \mathbf{x})\rVert^2\) with random \(\mathbf{x}\), \(\epsilon\), \(t\). (In the opposite convention, noise at \(t = 0\), the target is \(\mathbf{x} - \epsilon\).)
The true velocity at a point is an average over all data. Why is it fine to regress on the velocity of one pair?core
Mean squared error is minimized by the conditional mean of the targets. The single-pair targets at a point \(\mathbf{z}\) and time \(t\) have mean \(\E[\epsilon - \mathbf{x} \mid \mathbf{x}_t = \mathbf{z}]\), which is exactly the marginal velocity. Expanding the square, the CFM loss equals the FM loss plus a constant (the variance of the targets), so the gradients are identical (Lipman et al., Theorem 2). The same argument justifies \(\epsilon\)-prediction in DDPM.
If every training path is a straight line, why does the trained model need many steps?core
The straight training lines cross. At a crossing the network sees conflicting targets and learns their average. The resulting ODE cannot have crossing paths, so its paths bend: in the toy example of this session they are on average about 1.8 times longer than straight. Euler with few steps cuts corners on curved paths. Only the conditional paths are straight, not the learned marginal flow.
What does a single Euler step of a flow matching model produce, and why?core
From \(t = 1\), one step gives \(\epsilon - \mathbf{v}(\epsilon, 1)\). With data at \(t = 0\), \(\mathbf{v} = (\mathbf{z} - \hat{\mathbf{x}})/t\), so the result is \(\hat{\mathbf{x}}(\epsilon, 1)\), the best guess of the data from pure noise, which is the data mean. Every sample becomes the same blurry average. Few-step generation needs straighter flows (reflow) or distillation.
Is flow matching different from diffusion?core
With Gaussian paths \(\mathbf{x}_t = a_t\mathbf{x} + b_t\epsilon\), both learn the same kind of object, and the deterministic flow-matching ODE equals the probability-flow ODE of the corresponding diffusion. Velocity, noise and data predictions convert into each other (\(\hat{\mathbf{x}} = \mathbf{x}_t - t\hat{\mathbf{v}}\), \(\hat{\epsilon} = \mathbf{x}_t + (1 - t)\hat{\mathbf{v}}\) on the straight path), and their losses differ by time-dependent weights (229.12). The practical differences are the path (straight vs variance-preserving), the weighting and the timestep distribution. Flow matching also allows non-Gaussian sources and other couplings, which diffusion does not naturally do.
What is reflow, what does it guarantee, and what does it cost?advanced
Reflow generates pairs \((\Phi(\epsilon), \epsilon)\) by running the trained flow, then trains a new flow on these fixed pairs. Because ODE paths do not cross, the new pairs’ straight lines cross much less, so the new flow is straighter. Liu et al. prove that the marginals are preserved and that convex transport costs do not increase. Costs: a large synthetic dataset from many full ODE solves, a second training run, and inherited teacher errors, which can lower quality at many steps (Figure 4).
Why did SD3 sample training timesteps from a logit-normal distribution?core
Velocity prediction is hardest at intermediate noise levels: near data the answer is almost given, near pure noise the target is close to \(\epsilon\) minus the mean. Logit-normal sampling, \(t = \sigma(u)\) with \(u \sim \mathcal{N}(m, s^2)\), puts more training examples in the middle; with \(m = 0, s = 1\), 72.8% of timesteps fall in \([0.25, 0.75]\). Since log-SNR \(= -2\,\operatorname{logit}(t)\), it is a normal distribution on log-SNR centered at SNR = 1. In SD3’s comparison this setting had the best average rank.
Why shift timesteps toward noise at high resolution?advanced
Pixels of a high-resolution image are strongly correlated, so at the same noise level more global structure survives (averaging nearby pixels removes noise). A model then sees too few truly hard, high-noise examples, and at sampling the global layout gets fixed too late. The shift \(t' = \alpha t/(1 + (\alpha - 1)t)\) adds \(\log\alpha\) to the logit and divides the SNR by \(\alpha^2\). SD3 uses \(\alpha = 3\) at 1024², for training and sampling, chosen by a human preference study.
How does MM-DiT differ from a DiT with cross-attention?core
In cross-attention, image tokens attend to fixed text features; the text never updates. MM-DiT keeps separate weights for the text and image streams (norms, QKV, MLP) but concatenates both token sequences for one joint self-attention, so text tokens also read the image and both representations evolve layer by layer. The timestep and pooled text vector enter through adaLN modulation. SD3 reports better text understanding and typography with this design.
Your flow model gives good images at 50 steps; the product needs 4. What are your options?advanced
(1) A better solver and time grid: Heun or a higher-order solver, and a shifted grid that spends steps where the path curves most. Cheap, but 4 steps is usually not enough. (2) Reflow to straighten the flow, then fine-tune; costly data generation, some quality loss. (3) Distillation: progressive, consistency or adversarial distillation (FLUX.1 schnell uses latent adversarial diffusion distillation for 1 to 4 steps). (4) Also distill the guidance into the model to avoid two calls per step. Measure quality vs steps on held-out prompts.

Exercises

  1. Data \(x = 1\), noise \(\epsilon = 0.5\), \(t = 0.4\). Compute \(x_t\) and the rectified flow target. Then recover \(x\) from \(x_t\) and the true velocity.
  2. Show that one Euler step from \(t = 1\) with the exact marginal velocity returns \(\E[\mathbf{x}]\) for every noise sample. Use \(\mathbf{u}_t(\mathbf{z}) = (\mathbf{z} - \hat{\mathbf{x}}(\mathbf{z}, t))/t\).
  3. With shift \(\alpha = 3\), where does \(t = 0.5\) go? What are the SNR before and after?
  4. At \(t = 0.5\), how much larger is the velocity loss than the \(\epsilon\) loss for the same model? At \(t = 0.9\)?
  5. A sampler uses 28 steps with classifier-free guidance. How many network evaluations does one image cost? How many with 4 steps and guidance distilled into the model?
  6. In Lipman’s formula (229.13), set \(\sigma_\text{min} = 0\) and replace \(t\) by \(1 - t\). Show that you get formula (229.3), and that Lipman’s velocity \(\mathbf{x}_1 - \mathbf{x}_0\) is minus our velocity \(\epsilon - \mathbf{x}\).
Answerscheck
1. \(x_t = 0.6 \times 1 + 0.4 \times 0.5 = 0.8\); target \(\epsilon - x = -0.5\); \(\hat x = x_t - t v = 0.8 + 0.2 = 1.0\). 2. At \(t = 1\), \(\mathbf{x}_1 = \epsilon\) is independent of \(\mathbf{x}\), so \(\hat{\mathbf{x}}(\mathbf{z}, 1) = \E[\mathbf{x}]\) and \(\mathbf{u}_1(\epsilon) = \epsilon - \E[\mathbf{x}]\). One step with \(\Delta = 1\): \(\epsilon - (\epsilon - \E[\mathbf{x}]) = \E[\mathbf{x}]\). 3. \(t' = 1.5/2 = 0.75\). SNR goes from \((0.5/0.5)^2 = 1\) to \((0.25/0.75)^2 = 1/9\), that is divided by \(\alpha^2 = 9\). 4. By (229.12) the factor is \(1/(1 - t)^2\): 4 at \(t = 0.5\), 100 at \(t = 0.9\). 5. \(28 \times 2 = 56\) evaluations; 4 with distilled guidance. 6. With \(\sigma_\text{min} = 0\): \(\psi_s = (1 - s)\mathbf{x}_0 + s\mathbf{x}_1\) with \(\mathbf{x}_0\) noise and \(\mathbf{x}_1\) data. Put \(s = 1 - t\): \(t\,\epsilon + (1 - t)\mathbf{x}\), formula (229.3). Lipman’s velocity is \(d\psi_s/ds = \mathbf{x}_1 - \mathbf{x}_0 = \mathbf{x} - \epsilon\). Since \(ds/dt = -1\), \(d\mathbf{x}_t/dt = \epsilon - \mathbf{x}\): the same line, the velocity with the opposite sign.

Key takeaways

  • A flow model is an ODE (229.1): a network gives the velocity, and Euler steps (229.2) carry noise at \(t = 1\) to data at \(t = 0\).
  • Rectified flow trains on straight lines \(\mathbf{x}_t = (1 - t)\mathbf{x} + t\epsilon\) (229.3) with target \(\epsilon - \mathbf{x}\) (229.4, 229.8). Regressing on single pairs gives the same gradient as matching the averaged velocity (229.6, 229.9), but the loss has a floor.
  • Velocity, noise and data predictions are interchangeable (229.11); their losses differ only by time weights (229.12). Flow matching with Gaussian paths and diffusion are one family (229.10).
  • The learned flow curves because training lines cross. One step gives the data mean. Reflow (229.14) re-pairs noise with its own output, which straightens the flow and makes 1 to 2 steps work, at some cost in quality at many steps.
  • SD3: rectified flow, logit-normal timesteps (229.16) that focus on mid noise (a normal on log-SNR, 229.17), a resolution shift (229.18, \(\alpha = 3\) at 1024²), and the two-stream MM-DiT. FLUX.1 is a 12B rectified flow transformer.

Key formulas

\[ \mathbf{x}_t = (1 - t)\mathbf{x} + t\epsilon, \quad \mathcal{L} = \E\big\lVert \mathbf{v}_\theta(\mathbf{x}_t, t) - (\epsilon - \mathbf{x}) \big\rVert^2, \quad \mathbf{x}_{t - \Delta} = \mathbf{x}_t - \Delta\,\mathbf{v}_\theta(\mathbf{x}_t, t) \]
\[ \hat{\mathbf{x}} = \mathbf{x}_t - t\hat{\mathbf{v}}, \quad \hat{\epsilon} = \mathbf{x}_t + (1 - t)\hat{\mathbf{v}}, \quad t' = \frac{\alpha t}{1 + (\alpha - 1)t}, \quad \text{SNR}(t) = \left(\frac{1 - t}{t}\right)^2 \]

References for this session

  1. Lipman et al. (2022). Flow Matching for Generative Modeling
  2. Liu et al. (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
  3. Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
  4. Lipman et al. (2022). Flow Matching for Generative Modeling
  5. Liu et al. (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
  6. Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
  7. Albergo & Vanden-Eijnden (2022). Building Normalizing Flows with Stochastic Interpolants
  8. Liu, Zhang, Ma, Peng & Liu (2023). InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation
  9. Black Forest Labs (2024). FLUX.1 [schnell] model card
  10. Polyak et al. (Meta) (2024). Movie Gen: A Cast of Media Foundation Models
  11. Ho et al. (2020). Denoising Diffusion Probabilistic Models
  12. Song et al. (2020). Denoising Diffusion Implicit Models
  13. Song et al. (2020). Score-Based Generative Modeling through Stochastic Differential Equations
  14. Peebles & Xie (2022). Scalable Diffusion Models with Transformers
  15. Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models
  16. Ho & Salimans (2022). Classifier-Free Diffusion Guidance
  17. Song et al. (2023). Consistency Models