Diffusion, Score-Based Models & Flow Matching

A common discrete-time forward process (DDPM-style) is:

where $\bar\alpha_t = \prod_{s=1}^t \alpha_s$ and $\alpha_s = 1-\beta_s$ for a noise schedule $\beta_s$.

Equivalently, you can sample:

Ho et al. (2020) train with the noise-prediction loss, with $t$ drawn uniformly from $1,\dots,T$:

This is a reweighted form of the variational bound on the log-likelihood; in their experiments, dropping the weights gave better samples.

The score view connects denoising to the gradient of the log-density, $\nabla_x\log p_t(x)$ (Song and Ermon, 2019; Song et al., 2021). Under the Gaussian corruption above, the score of $q(x_t\mid x_0)$ is $-\epsilon/\sqrt{1-\bar\alpha_t}$, so a trained noise predictor gives a score estimate:

The score matching page derives why regressing on the noise recovers this score.

The network sees only $x_t$ and $t$, never the clean point or the noise that produced them, and many pairs $(x_0,\epsilon)$ lead to the same $x_t$. At a given $x_t$ the squared loss is therefore minimized by the average of their noise labels, weighted by how likely each clean point is given $x_t$, and that average gives the score of the noisy data distribution $p_t$:

The second equation is Tweedie's formula written for noise prediction; the score matching page gives it in terms of the clean point, with its source. It concerns the marginal $p_t$, not the Gaussian $q(x_t\mid x_0)$ of one clean point, whose score, $-\epsilon/\sqrt{1-\bar\alpha_t}$, is set by a single label. The best prediction is usually none of the labels it averages; the practice after the demo works this out for a distribution with two clean values.

Sampling: running the process backwards

Ho et al. (2020, Algorithm 2) start from $x_T\sim\mathcal N(0,I)$ and, for $t=T,\dots,1$, step

with $\sigma_t^2=\beta_t$ (one of two variances they found to work about equally well) and $z=0$ at the last step. This needs one network evaluation per step, and $T=1000$ in their experiments.

• Fewer steps. DDIM (Song, Meng and Ermon, 2021) reuses the same trained network with a deterministic update. It estimates the clean sample, $\hat x_0 = \big(x_t-\sqrt{1-\bar\alpha_t}\,\epsilon_\theta(x_t,t)\big)/\sqrt{\bar\alpha_t}$, and jumps to an earlier time $s<t$ with $x_s=\sqrt{\bar\alpha_s}\,\hat x_0+\sqrt{1-\bar\alpha_s}\,\epsilon_\theta(x_t,t)$. Skipping steps this way gave samples 10 to 50 times faster than DDPM sampling in their experiments.
• Continuous time. Song et al. (2021) treat the noising process as a stochastic differential equation. Its time reversal, or a deterministic "probability flow" ODE with the same marginal distributions, can be integrated with standard numerical solvers using the learned score.
• Guidance. To sample from a conditional model, for example one conditioned on a text prompt $c$, classifier-free guidance (Ho and Salimans, 2021) trains one network both with $c$ and with $c$ dropped, then samples with $\tilde\epsilon = (1+w)\,\epsilon_\theta(x_t,c) - w\,\epsilon_\theta(x_t)$. A guidance weight $w>0$ trades sample diversity for closer agreement with $c$.
• Latent diffusion. Rombach et al. (2022) run the whole process in the latent space of a pretrained autoencoder instead of on pixels, which makes high-resolution generation much cheaper.

Flow matching trains a velocity field instead of a noise predictor. Lipman et al. (2023) show that the diffusion noising paths are one family among the probability paths it can use.