Normalizing Flows: Tractable Density via Invertible Transforms

Change of variables

Let $x=f(z)$ where $f$ is invertible and $z\sim p_z$. Then:

so:

The determinant term is the volume correction. Relative to the base density at the corresponding latent point, an inverse map that squeezes a small data-space volume into a smaller latent-space volume multiplies density by a factor below one; an inverse map that expands that volume multiplies it by a factor above one.

Equivalently, expansion by the forward map spreads probability over more data-space volume and lowers density. For example, in one dimension x = 2z gives p_x(x) = p_z(x / 2) / 2. When comparing different points or transforms, the base-density term can also change, so the determinant alone does not determine which total density is larger.

This is the tractable-density advantage of flows, but also the constraint: a flexible transform is useful for likelihood work only when the needed map direction and log-determinant stay tractable.

The volume term does not rank densities by itself

For a one-dimensional affine map $x=az+b$ with $a\neq0$ and a standard normal base, the change of variables reads

Compare map A, $x=2z$, with map B, $x=z/2$, at the point $x=1$. B's volume term is larger, $+\log 2$ against $-\log 2$. But B must start from $z=2$, two standard deviations out, while A starts from $z=1/2$:

The same happens with $C$, $x=3z+1$, and $D$, $x=z/3+1$, at $x=2$: D's volume term is larger by $2\log 3$, yet $\log p_C(2)-\log p_D(2)=\tfrac{40}{9}-2\log 3\approx2.247$. The Jacobian determinant matters only together with the base density at the preimage.

The absolute value ignores orientation. The map $x=-2z$ has the same volume term as $x=2z$, and at $x=1$ the same density, because its preimage $z=-1/2$ has the same standard normal density as $1/2$. With a base distribution that is not symmetric, the two densities can differ.

Maps that are not one-to-one

The formula needs exactly one preimage. For $x=z^2$, both $z=\sqrt x$ and $z=-\sqrt x$ reach a point $x>0$, and each contributes its base density divided by $|dx/dz|=2\sqrt x$:

One branch alone integrates to $1/2$. At $x=0$ the derivative vanishes, and below $0$ there is no preimage. Normalizing flows avoid these cases by building $f$ from invertible layers whose Jacobian determinants are never zero.

Composition of flows

If $f=f_K\circ\cdots\circ f_1$ maps latent variables forward, with $z_k=f_k(z_{k-1})$ and $z_K=x$, then the inverse-Jacobian log-determinants add as a negative sum of forward log-determinants:

This follows from the determinant of a product of Jacobians. A general $d\times d$ determinant costs $O(d^3)$ operations, so practical flows are built from layers with cheap Jacobians. The coupling layers of Real NVP (Dinh et al., 2017), for example, have triangular Jacobians, whose log-determinant is just the sum of the logs of the $d$ diagonal entries. Papamakarios et al. (2021) review the main designs.

A one-dimensional check of the sum: following $z_1=2z_0$ by $x=z_1/4$ gives the single map $x=z_0/2$, and the forward log-determinants $\log 2+\log\tfrac14=-\log 2$ equal that map's.