Legacy Concept Lab

Normalizing Flows: Exact Likelihood via Invertible Transforms

Flows provide exact likelihood (unlike GANs) and exact sampling (unlike EBMs)—the "best of both worlds"

Concept 39 of 100Generative ModelsPhase 4

#39FlowsGenerative Models

key equation\log p_x(x) = \log p_z(f^{-1}(x)) + \log \left| \det \frac{\partial f^{-1}}{\partial x} \right|

Phase 4: Generative modeling familiesConcept 39 of 100

Why It Matters for Modern Models

Flows provide exact likelihood (unlike GANs) and exact sampling (unlike EBMs)—the "best of both worlds"
The mathematical foundation for flow matching/rectified flows which are replacing traditional diffusion
Understanding Jacobian determinants and invertibility constraints illuminates architectural design choices

What is still poorly explained in textbooks and papers:

The challenge is making f invertible AND having tractable Jacobian determinant—this drives architecture choices (coupling layers, autoregressive flows)
Unlike VAEs, no variational bound: you get exact log p(x), but at the cost of architectural constraints
Modern flow matching avoids the Jacobian entirely by learning velocity fields—converges to OT map

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation

\log p_x(x) = \log p_z(f^{-1}(x)) + \log \left| \det \frac{\partial f^{-1}}{\partial x} \right|

Normalizing flows transform a simple base distribution $p_z(z)$ through an invertible function $f$ :

x = f(z), \quad z \sim p_z

The change of variables formula gives exact log-likelihood:

\log p_x(x) = \log p_z(f^{-1}(x)) + \log \left| \det \frac{\partial f^{-1}}{\partial x} \right|

Composition of flows: $f = f_K \circ \cdots \circ f_1$ with log-det Jacobian summing:

\log p_x(x) = \log p_z(z_0) + \sum_{k=1}^K \log \left| \det \frac{\partial f_k}{\partial z_{k-1}} \right|

Rezende & Mohamed2015ICML

Kingma & Dhariwal2018NeurIPS

Explore this concept from different angles — like a mathematician would.