Variational Autoencoders

Latent variable likelihood

For one observed datapoint $x$, assume a prior $p(z)$ and a decoder likelihood $p_\theta(x\mid z)$. The latent-variable model, marginal likelihood, and true posterior are

$$\begin{aligned} p_\theta(x,z)&=p(z)p_\theta(x\mid z),\\ p_\theta(x)&=\int p_\theta(x,z)\,dz,\\ p_\theta(z\mid x)&=\frac{p_\theta(x,z)}{p_\theta(x)}. \end{aligned}$$

The evidence $p_\theta(x)$ appears in the denominator, so exact posterior inference and exact likelihood training are tied to the same difficult integral. The claim-bearing VAE move is summarized by

$$\begin{aligned} \log p_\theta(x) &= \mathcal L(\theta,\phi;x) + \mathrm{KL}\!\left(q_\phi(z\mid x)\,\|\,p_\theta(z\mid x)\right),\\ \mathcal L(\theta,\phi;x) &= \mathbb E_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)] - \mathrm{KL}\!\left(q_\phi(z\mid x)\,\|\,p(z)\right),\\ \epsilon&\sim\mathcal N(0,I), \qquad z=\mu_\phi(x)+\sigma_\phi(x)\odot\epsilon. \end{aligned}$$

The first line says the ELBO gap is exactly the KL from the learned encoder $q_\phi(z\mid x)$ to the true posterior. The second line separates reconstruction likelihood from prior pressure. The third line keeps randomness in external noise $\epsilon$, so gradients of sampled reconstruction terms can flow through $z$ into the encoder outputs.

The ELBO identity

Let $q_\phi(z\mid x)$ be any approximate posterior whose support is compatible with the true posterior. To keep the equations readable, write $q(z)$ for $q_\phi(z\mid x)$ and $\mathbb E_q$ for expectation under that distribution:

$$\mathbb E_q[\cdot] \equiv \mathbb E_{q(z)}[\cdot].$$

Using Bayes' rule and rearranging gives the exact identity

$$\log p_\theta(x)=\mathcal L+G.$$

Here $\mathcal L$ is the ELBO and $G$ is the inference gap. Let $p_*(z)$ denote the true posterior $p_\theta(z\mid x)$. Then the gap is

$$G=\mathrm{KL}(q(z)\,\|\,p_*(z)).$$

Equivalently,

$$G=\mathbb E_q[\log q(z)-\log p_*(z)].$$

The ELBO term is

$$\mathcal L = \mathbb E_q[\log p_\theta(x,z)-\log q(z)].$$

The gap can also be read by subtraction:

$$G=\log p_\theta(x)-\mathcal L.$$

KL is nonnegative, so

$$\mathcal L(\theta,\phi;x)\le \log p_\theta(x).$$

Maximizing the ELBO improves this lower bound. The bound is tight exactly when the encoder distribution equals the true posterior, up to probability-zero events.

Reconstruction likelihood plus prior KL

Because

$$\log p_\theta(x,z) = \log p_\theta(x\mid z)+\log p(z),$$

the ELBO can be rewritten as

$$\begin{aligned} \mathcal L &= \mathbb E_q[\log p_\theta(x\mid z)] \\ &\quad- \mathrm{KL}(q(z)\,\|\,p(z)). \end{aligned}$$

The first term is a reconstruction log likelihood, not generically a pixel distance. The second term keeps the encoder's posterior close to the prior used for generation. At generation time we sample $z\sim p(z)$, not $z\sim q_\phi(z\mid x)$ for a training example, so the prior match is load-bearing.

Reconstruction likelihood is not always MSE

The reconstruction term is

$$\mathbb E_q[\log p_\theta(x\mid z)].$$

It becomes a familiar loss only after choosing a decoder distribution. With a fixed-variance Gaussian decoder centered at $f_\theta(z)$, the negative log likelihood is a scaled squared error plus a constant. So MSE appears as a Gaussian negative log likelihood only after that modeling choice.

If $x$ is binary and

$$p_\theta(x\mid z)=\operatorname{Bernoulli}(\pi_\theta(z)),$$

then the negative reconstruction log likelihood is binary cross-entropy.

Reparameterization trick

For a diagonal Gaussian encoder, the network outputs $\mu_\phi(x)$ and $\sigma_\phi(x)$. We sample with external noise:

$$\epsilon\sim\mathcal N(0,I), \qquad z=\mu_\phi(x)+\sigma_\phi(x)\odot\epsilon.$$

This keeps the randomness in $\epsilon$ and makes the sample differentiable with respect to the encoder outputs $\mu_\phi(x)$ and $\sigma_\phi(x)$.

Beta-VAE objective

A beta-VAE changes the training objective to

$$\begin{aligned} \mathcal L_\beta &= \mathbb E_q[\log p_\theta(x\mid z)] \\ &\quad- \beta\,\mathrm{KL}(q(z)\,\|\,p(z)). \end{aligned}$$

When $\beta=1$, this is the standard VAE ELBO. When $\beta\ne 1$, it is a modified objective rather than the original ELBO. For $\beta>1$, it remains a lower bound because it subtracts extra nonnegative KL penalty, but it is no longer the standard maximum-likelihood ELBO. For $\beta<1$, it is generally not guaranteed to lower-bound $\log p_\theta(x)$.

Larger $\beta$ puts more pressure on $z$ to carry less information and stay close to the prior. This can encourage more factor-like representations in some settings, but it does not guarantee disentanglement. If the pressure is too strong, the encoder may approach $q_\phi(z\mid x)\approx p(z)$, so $z$ carries little information about $x$.