Variational Inference

Let $x$ be observed data and let $z$ be latent. A latent-variable model specifies a joint density or mass function

$$p_\theta(x,z)=p_\theta(x\mid z)p(z),$$

where $\theta$ are model parameters. The exact posterior is

$$p_\theta(z\mid x)=\frac{p_\theta(x,z)}{p_\theta(x)}.$$

The evidence is the normalizer

$$p_\theta(x)=\sum_z p_\theta(x,z),$$

or an integral when $z$ is continuous. This sum is often the hard object.

Choose a variational distribution $q(z)$ from a family $\mathcal Q$. The ELBO is

$$\mathcal L(q) = \mathbb E_{q(z)} \left[ \log p_\theta(x,z)-\log q(z) \right].$$

Equivalently,

$$\mathcal L(q) = \mathbb E_q[\log p_\theta(x,z)] + H(q),$$

where $H(q)=-\mathbb E_q[\log q(z)]$ is entropy. A good variational distribution must put mass on high-joint-probability latent states, but it is also rewarded for keeping enough entropy when that helps the bound.

Now derive the gap. Start from the reverse KL to the exact posterior:

$$\mathrm{KL}\big(q(z)\,\|\,p_\theta(z\mid x)\big) = \mathbb E_q \left[ \log q(z)-\log p_\theta(z\mid x) \right].$$

Use $\log p_\theta(z\mid x)=\log p_\theta(x,z)-\log p_\theta(x)$:

$$\begin{aligned} \mathrm{KL}\big(q\,\|\,p_\theta(z\mid x)\big) &= \mathbb E_q[\log q(z)-\log p_\theta(x,z)+\log p_\theta(x)] \\ &= \log p_\theta(x) - \mathbb E_q[\log p_\theta(x,z)-\log q(z)] \\ &= \log p_\theta(x)-\mathcal L(q). \end{aligned}$$

So

$$\log p_\theta(x) = \mathcal L(q) + \mathrm{KL}\big(q\,\|\,p_\theta(z\mid x)\big).$$

Because KL divergence is nonnegative,

$$\mathcal L(q)\le \log p_\theta(x).$$

This is also Jensen's inequality in disguise:

$$\log p_\theta(x) = \log \sum_z q(z)\frac{p_\theta(x,z)}{q(z)} \ge \sum_z q(z)\log \frac{p_\theta(x,z)}{q(z)} = \mathcal L(q),$$

assuming $q(z)>0$ only where the ratio is well-defined.

Mean-field variational inference chooses a factorized family such as

$$q(z_1,\ldots,z_m)=\prod_{j=1}^m q_j(z_j).$$

This can make coordinate updates tractable: optimize one factor $q_j$ while holding the others fixed. The cost is representational. If the posterior has strong dependence or multiple separated modes, a simple $q$ family may fit only one part of it.

The direction of KL matters. The usual ELBO identity minimizes $\mathrm{KL}(q\,\|\,p)$, often called exclusive or reverse KL in this context. With a restricted family, it can prefer a sharp approximation around one mode rather than spreading mass through a low-probability region between modes. That is a feature of the objective, not a bug in the arithmetic.

Amortized variational inference changes how $q$ is produced. Instead of optimizing separate variational parameters for every $x$, a network predicts them:

$$q_\phi(z\mid x).$$

That is the bridge to variational autoencoders. The general idea here is still the same: optimize an ELBO, inspect the KL gap, and remember what the variational family can and cannot express.