Score Matching & Score-Based Generative Models

The score function and score matching

For a non-normalized model $p_\theta(x)=q_\theta(x)/Z(\theta)$, the data-gradient score ignores the constant $Z(\theta)$:

Hyvärinen (2005) showed that this objective, which needs only the model's score and its derivatives, has the same minimizer as matching the unknown data score. It avoids the normalizing constant, but the trace term needs every diagonal entry of the Jacobian of $s_\theta$, which costs about one backward pass per input dimension for a neural network.

Denoising score matching (diffusion-friendly)

Add Gaussian noise. The conditional target, noise-prediction scale, and reverse-time dynamics share the same learned score field:

Denoising score matching trains:

So predicting noise $\epsilon_\theta$ is equivalent (up to scaling) to predicting the score:

In the reverse-time equations, the forward noising process is $dx=f(x,t)\,dt+g(t)\,dw$, $\bar w$ is a Brownian motion run backwards in time, and $dt$ is negative when integrating from $t=T$ down to $0$ (Song et al., 2021). The first is a stochastic sampler; the second, the probability-flow ODE, is deterministic and has the same marginal distributions at every $t$.

The conditional DSM target $\nabla_{\tilde x}\log q(\tilde x\mid x)$ is the label for one known clean source; the marginal score field $\nabla_{\tilde x}\log p_\sigma(\tilde x)$ averages over possible clean sources. Regression closes that gap. At each $\tilde x$ the DSM loss is minimized by the average label, and that average is exactly the marginal score:

This is Tweedie's formula (Efron, 2011). It is why training on noisy copies of known clean points (Vincent, 2011) recovers the score of the noisy data distribution, and why an optimal denoiser, which outputs $\mathbb E[x\mid\tilde x]$, determines the score.

A one-dimensional check: explicit and implicit objectives

Take data $X\sim\mathcal N(0,v)$, where $v$ is the variance, and the unnormalized model $q_\theta(x)=\exp(-\theta x^2/2)$ with $\theta>0$. Its normalizing constant $Z(\theta)=\sqrt{2\pi/\theta}$ drops out of the score, $s_\theta(x)=-\theta x$, while the data score is $s_*(x)=-x/v$. Index both objectives by the variance:

$J_v$ is half the Fisher divergence and needs the unknown data score; $L_v$ is the objective $\mathcal L_{SM}$ above, which needs only the model. Expanding the square in $J_v$ leaves one term with $s_*$, and integration by parts removes it: $-\mathbb E[s_\theta s_*]=-\int s_\theta\,p_*'\,dx=\mathbb E[\partial_x s_\theta]$, provided $p_*(x)s_\theta(x)\to0$ as $|x|\to\infty$, as it does here (Hyvärinen, 2005). What is left over is a constant that depends on the data but not on $\theta$:

Both objectives are minimized at $\theta=1/v$, where the model matches the data. There $J_v=0$ but $L_v=-1/(2v)$: the implicit objective can be negative, and its value is not a distance. The two share a minimizer, not their values.