Flow Matching & Rectified Flows

Generative flow and regression target

We model a trajectory $x_t$ with a neural velocity field $v_\theta$, then train that field to match a target velocity $u_t$:

Here $t$ is drawn uniformly from $[0,1]$, $x_0$ from the noise distribution and $x_1$ from the data. Regressing on these per-pair (conditional) targets has the same gradients as regressing on the marginal field described below, which cannot be computed directly. Lipman et al. (2023) prove this for paths conditioned on the data point $x_1$, and Tong et al. (2024) for paths conditioned on a pair $(x_0,x_1)$, as here.

Rectified flow (straight paths)

Choose a straight interpolation between noise $x_0$ and data $x_1$. Differentiating the path gives the supervised velocity label:

The remaining displacement $x_1-x_t$ is shorter than the training label except at $t=0$.

So the useful identity is:

For a single straight conditional pair, the velocity label stays fixed as $t$ changes; the remaining integration time shrinks.

What the network learns from many pairs

At each point and time, the squared loss is minimized by the conditional mean of the label:

Liu et al. (2023) call the ODE driven by $v^\star$ the rectified flow. It carries the noise distribution onto the data distribution, but its paths are generally not straight: where straight pair-paths cross, $v^\star$ averages their directions, and the ODE paths bend instead of crossing. Their reflow step retrains on (noise, sample) pairs produced by the previous flow, which straightens the paths and allows fewer integration steps.

Which pairs are used also matters. With independent pairs, each $x_0$ drawn without regard to $x_1$, the learned flow moves noise to data but is not in general the optimal-transport map between the two distributions. The "optimal transport" path in Lipman et al. (2023) refers to each conditional straight path, not to an optimal pairing of the distributions. Tong et al. (2024) pair noise and data within each minibatch by solving a small optimal-transport problem (OT-CFM); this gives straighter flows that are cheaper to integrate, and it approximates dynamic optimal transport when the true optimal pairing is available.

A different coupling: independent Gaussian endpoints

The equations above hold for any pairing of $x_0$ with $x_1$. The first Python example below fixes one: the one-dimensional affine pairing $x_1=2x_0+1$, for which $x_t=(1+t)x_0+t$. For each $t\in[0,1]$ that map is invertible in $x_0$, since $1+t\ge1$, so a state and time determine the pair and its label, $u=(x_t+1)/(1+t)$. The demo instead pairs synthetic two-dimensional Gaussian noise points with two-moons data points. Now take a one-dimensional coupling that is neither: draw the endpoints independently, $X_0\sim\mathcal N(0,1)$ and $X_1\sim\mathcal N(0,4)$, where 4 is the target variance and 2 its standard deviation. Keep the straight interpolation $X_t=(1-t)X_0+tX_1$ for $t\in[0,1]$ and the paired label $U=X_1-X_0$. Many pairs now pass through the same point at the same time, each with its own label. Neither earlier example is wrong; each uses a different coupling.

Here $q$ is a variance, with minimum $4/5$ at $t=1/5$; the standard deviation is $s(t)=\sqrt{q(t)}$, with minimum $\sqrt{4/5}\approx0.894$. Because $(U,X_t)$ is jointly Gaussian with mean zero, conditioning on $X_t=x$ gives the marginal field and the spread of the labels compatible with that point:

using $5q(t)-(5t-1)^2=4$. Take the pair $(x_0,x_1)=(1,3)$. At $t=0.5$ its state is 2, its label 2, its remaining displacement 1, and $u(2,0.5)=2.4$. At $t=0.2$ its state is 1.4 and its label is still 2, but $u(1.4,0.2)=0$, with label variance 5. The field is the mean over pairs through that point, not over all pairs. The mean is exact; the spread is variation among compatible labels, not uncertainty about the mean.

Following the field, $dz/dt=u(z,t)$, has the exact solution $Z_t=s(t)Z_0$, because $s'(t)/s(t)=q'(t)/(2q(t))=(5t-1)/q(t)$. So $Z_t$ and $X_t$ have the same law $\mathcal N(0,q(t))$ at every time: the one-dimensional case of the marginal-preserving flow of Liu, Gong and Liu (2023, §3.1, Theorem 3.3). They are different paths with different couplings. The flow pairs $Z_1=2Z_0$, whereas training drew $X_1$ independently of $X_0$, so the flow's destination is not the training partner of its start: a drawn pair has $X_1=2X_0$ with probability zero. For $Z_0>0$ the flow moves toward zero until $t=0.2$ and away from it afterwards; it is not affine in time, although every training path is.

At $t=1$ two conditionings differ. Given $X_1=x_1$, the state $X_t$ has variance $(1-t)^2$, and the pair-conditional field $(x_1-x)/(1-t)$ is undefined at $t=1$; it is not evaluated or clamped there. The label $U$ is still defined, and conditioning on $X_1=x$ gives $u(x,1)=x$ with label variance 1. The marginal field is regular on all of $[0,1]$, and one endpoint has no weight in the uniform-time integrals below. Lipman et al.'s Gaussian conditional paths keep a positive endpoint width (their Eqs. 20–23), so this zero-width endpoint rests on the calculation here, not on their construction.

Following a supplied field

A trained model supplies some field. Here the field is chosen instead: $v_\varepsilon(x,t)=u(x,t)+\varepsilon x$ for a fixed $\varepsilon$, a stipulated error rather than the result of training. Its exact trajectory from $z_0=x_0$ is $e^{\varepsilon t}s(t)x_0$, which ends at $2e^{\varepsilon}x_0$. Explicit Euler with $N$ uniform steps, $h=1/N$ and $t_k=k/N$, evaluates the field at its own numerical state:

The reference endpoint is $2x_0$, the exact flow of the true field $u$. The endpoint discrepancy splits into signed parts:

With $x_0=1$ and $\varepsilon=0$ the field is exact and only numerical error remains: $N=8$ ends at 1.659138 and $N=16$ at 1.822732, against 2. With $\varepsilon=0.2$ the supplied field's exact endpoint is 2.442806, a field-induced error of +0.442806. The numerical errors are −0.443552 at $N=8$ and −0.234936 at $N=16$, so the totals are −0.000746 and +0.207870. Refinement reduced the numerical error, as it should, and in doing so removed error that had been cancelling the field's. Signed errors add; absolute distances do not, since $0.443552+0.442806\ne0.000746$. None of this makes eight steps better in general. With $\varepsilon=-0.2$ both errors are negative and nothing cancels.

A separate counterexample: with $\varepsilon=0$ and $N=1$, the single step is $z_1=x_0+u(x_0,0)=x_0-x_0=0$ for every seed. Every training path is straight, yet one Euler step of the marginal field collapses every sample to zero. Straight conditional paths do not by themselves make one-step sampling accurate.

The field is linear in $x$, so $z_N=m_Nx_0$ for a number $m_N$ and the sampler's output law is $\mathcal N(0,m_N^2)$. Its exact 2-Wasserstein distance to the target $\mathcal N(0,4)$ is $\big||m_N|-2\big|$, including when $m_N<0$ reflects every seed. That is a population distance for this Gaussian family. It is not the error of one seed, and it is not an image-quality measure.

What the training loss can and cannot see

Use the unhalved squared loss with $t$ uniform on $[0,1]$. Conditioned on one state and time, the expected loss of the supplied field against the paired labels is

Only after averaging over $X_t\sim\mathcal N(0,q(t))$ does $x^2$ become $q(t)$. With $\int_0^1q\,dt=5/3$ and $\int_0^1 4/q\,dt=2\big[\arctan\tfrac{5t-1}{2}\big]_0^1=2\big(\arctan2+\arctan\tfrac12\big)=\pi$:

These are exact population values for this family; they match the equal-gradient statement of Lipman et al.'s Theorem 2. Their printed proof assumes bounded fields, which a linear field is not, so the equality here rests on the explicit calculation, in which every second moment is finite. The floor $\pi$ is the spread of labels compatible with a state, not measurement noise, numerical error or an unavoidable endpoint error. Individual sample gradients need not agree: for the pair (1, 3) at $t=0.2$, the CFM sample derivative at $\varepsilon=0$ is $2(u-U)x=-28/5$, while the FM sample derivative and both population derivatives are zero. Neither loss depends on $N$. Neither is an average along a generated trajectory or over one seed.

The floor belongs to the coupling. The dependent coupling $X_1=2X_0$ has the same endpoint marginals but $X_t=(1+t)X_0$, field $u(x,t)=x/(1+t)$ and zero label variance, so its CFM floor is 0 and its intermediate law differs. It is also distinct from this page's first Python example, $x_1=2x_0+1$. Endpoint data alone do not determine one field, and raw CFM losses under different couplings do not rank model quality.

Source statements, derivations and judgment

Source statements. Lipman et al. (2023; arXiv:2210.02747v2): Eq. 8 aggregates conditional fields into a marginal field, Eq. 9 defines the CFM objective, and Theorem 2 states that the FM and CFM losses differ by a parameter-independent constant and have equal gradients. In §6.2 and Figure 7 they measure the per-pixel mean squared error of ImageNet-32 solutions at up to 100 function evaluations against 1,000-evaluation solutions over 256 seeds, using the midpoint method for that numerical error, and they report FID separately. Liu, Gong and Liu (2023; arXiv:2209.03003v1): §2.1, Eq. 1 regresses $X_1-X_0$ along interpolated pairs and allows an independent coupling. §3.1, Definitions 3.1–3.2 and Theorem 3.3 define the expected-velocity field and its marginal-preserving flow. §5 uses Euler and adaptive RK45, Table 1 counts function evaluations as network calls, and Figure 8 varies the number of Euler steps against image metrics.

Derivations here. The Gaussian field, label variance, exact flows, Euler values, loss identities and Wasserstein distance above are this page's own calculations for one chosen family; neither paper reports these numbers. Nothing here reproduces either paper's sampler, image quality, speed or memory measurements, and nothing here trains a model or performs reflow.

Judgment. For a trained model the true field and flow are unknown. Freezing the field, seeds and numerical precision, then comparing a refinement or an independently implemented solver, measures the numerical sensitivity of integrating that field. Lipman et al.'s 1,000-evaluation comparison is of this kind, and a high-evaluation reference is itself numerical, not ground truth. A separately specified criterion on target data, such as FID for images or the exact distance above, evaluates the output distribution, where field and solver errors combine. Neither comparison alone identifies every error. Held-out CFM risk differences between two fields cancel the unknown floor in expectation only under the same coupling, path and time law; finite-sample uncertainty remains, and a smaller field risk need not give better samples.