Energy-Based Models & Score Functions

Why It Matters for Modern Models

An energy-based model assigns a scalar energy E(x) to each input and gives it probability proportional to exp(−E(x)), so lower energy means higher probability. Because normalization is done separately, any network that outputs one number can supply the energy; the cost is that the normalizer is generally intractable for high-dimensional inputs, which makes training and sampling hard (Song & Kingma, 2021).
A classifier can be read as an energy-based model: with logits f_θ(x), set E_θ(x, y) = −f_θ(x)[y]. The softmax then gives p(y | x), and the energy of x alone is minus the log-sum-exp of the logits, which defines an unnormalized p(x) (Grathwohl et al., 2020).
Score-based diffusion models learn the score ∇ₓ log pₜ of the noised data at each noise level directly. For an energy-based model the score is −∇ₓE, and neither depends on the normalizer, which is why score matching can train a model without ever computing Z.

Questions to Think About

Adding a constant to the energy leaves every probability unchanged, because Z changes by the same factor; why does that mean only differences in energy carry meaning?
Why is sampling from an energy-based model with MCMC slow in high dimensions, and how do diffusion models avoid long chains by learning to reverse a fixed noising process instead?

Core Math

Key Equation

p_\theta(x) = \frac{\exp(-E_\theta(x))}{Z_\theta}

An energy-based model gives each input $x$ a scalar energy $E_\theta(x)$ and turns it into a probability by exponentiating and normalizing:

p_\theta(x) = \frac{\exp(-E_\theta(x))}{Z_\theta}, \quad Z_\theta = \int \exp(-E_\theta(x))\, dx

Low energy means high probability. The normalizer $Z_\theta$ , called the partition function, is an integral over every possible input and is intractable for high-dimensional data such as images or text.

The model's score, the gradient of the log-density with respect to the input, does not involve $Z_\theta$ :

s_\theta(x) = \nabla_x \log p_\theta(x) = -\nabla_x E_\theta(x)

The gradient of the negative log-likelihood, averaged over the data, is a difference of two expectations:

\nabla_\theta \mathcal{L} = \mathbb{E}_{x \sim p_{\text{data}}}[\nabla_\theta E_\theta(x)] - \mathbb{E}_{x \sim p_\theta}[\nabla_\theta E_\theta(x)]

The first term lowers the energy of the data; the second raises the energy of samples from the model. The second term needs samples from $p_\theta$ , usually drawn with Markov chain Monte Carlo (MCMC). Contrastive divergence approximates it with only a few MCMC steps started at the data (Hinton, 2002); score matching and noise-contrastive estimation avoid sampling from the model altogether (Song & Kingma, 2021).