Continuous Function

Generative Physics

Diffusion, flow matching, and the geometry of generation

Generation as Transport from Noise to Data

How do we generate samples from a complicated distribution, such as the distribution of natural images? Diffusion and flow-matching models define a continuous path of distributions from simple noise to the data, and train a network to move samples along it.

Score-based diffusion, its deterministic form (the probability flow ODE), and flow matching can all be written as a differential equation that carries a noise sample to a data sample (Song et al., 2021; Lipman et al., 2023). They differ in the path they choose and in the quantity the network learns. Chapter 20 of Bishop & Bishop (2024) gives a textbook treatment of diffusion models.

Score Functions

The score of a distribution is the gradient of its log-probability:

Score Definition

s(x) = ∇x log p(x)

The score points toward higher probability. Following it alone climbs to a mode, a local peak of p, as the paths in the figure do. Adding noise of the right size at each step, as in Langevin dynamics, turns this climb into a way of drawing samples from p (Song & Ermon, 2019).

The figure shows the score field for a mixture of two Gaussians. Midway between them, at (0, 0), the score is zero.

Notebook: Score Matching →

Diffusion Models

Diffusion models define a forward process that gradually adds noise to data, and learn to reverse it (Ho et al., 2020). In continuous time, a forward process dx = f(x, t)dt + g(t)dw has a reverse-time counterpart (Anderson, 1982; Song et al., 2021):

Reverse SDE

dx = [f(x,t) - g(t)²∇log pt(x)]dt + g(t)dw̄

Here time runs backwards from noise to data, w̄ is a Brownian motion running in reverse time, and pt is the distribution of noised data at time t. Since f and g are chosen in advance, the only unknown is the score ∇log pt, so a network that estimates it at every noise level is enough to run the process backwards and generate samples.

Flow Matching

Flow matching learns a velocity field directly (Lipman et al., 2023; Liu et al., 2023). Training draws a noise sample x0 and a data sample x1, places a point on the straight line between them, and asks the network to predict that line's velocity:

Straight-line conditional path

xt = (1-t)x0 + tx1
target velocity: x1 - x0

Each training pair moves in a straight line, but the learned field is not that line. Many pairs pass through the same point xt, and the network learns their average, E[x1 − x0 | xt]. With noise and data paired at random, the samples generated by integrating that field generally follow curved paths. Rectified flow (Liu et al., 2023) re-pairs each noise sample with the output the trained flow produces from it and trains again; each round makes the paths straighter, so fewer integration steps are needed. Pairing noise and data by minibatch optimal transport instead of at random also gives simpler flows and faster sampling (Tong et al., 2024).

The Big Picture

All these methods share a common structure:

  • → Define a path between noise and data
  • → Learn a vector field along that path
  • → Integrate it to generate samples

The differences lie in the choice of path and the training target. Score matching learns the gradient of the log-probability; flow matching regresses on velocities. Lipman et al. (2023) show that the usual diffusion paths are special cases of flow matching with Gaussian probability paths.

Sources

  1. Bishop, C. M., & Bishop, H. (2024). Deep Learning: Foundations and Concepts, chapter 20: Diffusion Models. Springer. doi.org/10.1007/978-3-031-45468-4
  2. Song, Y., & Ermon, S. (2019). Generative Modeling by Estimating Gradients of the Data Distribution. NeurIPS. arxiv.org/abs/1907.05600
  3. Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS. arxiv.org/abs/2006.11239
  4. Anderson, B. D. O. (1982). Reverse-time diffusion equation models. Stochastic Processes and their Applications 12(3). doi.org/10.1016/0304-4149(82)90051-5
  5. Song, Y., et al. (2021). Score-Based Generative Modeling through Stochastic Differential Equations. ICLR. arxiv.org/abs/2011.13456
  6. Lipman, Y., et al. (2023). Flow Matching for Generative Modeling. ICLR. arxiv.org/abs/2210.02747
  7. Liu, X., Gong, C., & Liu, Q. (2023). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. ICLR. arxiv.org/abs/2209.03003
  8. Tong, A., et al. (2024). Improving and generalizing flow-based generative models with minibatch optimal transport. Transactions on Machine Learning Research. arxiv.org/abs/2302.00482