Variational Autoencoders

A latent-variable model trained by maximizing the evidence lower bound (ELBO). The bound falls short of the log-likelihood by the KL divergence from the encoder to the true posterior, so the encoder learns approximate inference, not just compression.

Advanced · Undergraduate mathematics · about 20 minutes

Reading map and next steps

Intuition

The idea in plain words, before the symbols.

Maximum likelihood asks for a model that assigns high log probability to the observed data:

\log p_\theta(x).

A variational autoencoder (Kingma and Welling, 2014; Rezende, Mohamed and Wierstra, 2014) is a maximum-likelihood model with a hidden cause. It assumes that each observation $x$ was produced from an unobserved latent variable $z$ :

\begin{aligned} z &\sim p(z), \\ x &\sim p_\theta(x\mid z). \end{aligned}

Training would be easy if we could compute the marginal likelihood

\begin{aligned} p_\theta(x) &= \int p_\theta(x,z)\,dz. \end{aligned}

The problem is that this integral is usually hard for neural decoders. Bayesian inference tells us what we would like to use:

p_\theta(z\mid x).

But that posterior contains the same hard evidence term $p_\theta(x)$ . The VAE move is to learn an encoder

q_\phi(z\mid x)

that acts as a tractable approximate posterior.

The central mechanism is not "an autoencoder with noise." It is: replace the intractable posterior with a learned distribution $q_\phi(z\mid x)$ , then optimize a lower bound whose looseness is exactly a KL divergence from $q_\phi$ to the true posterior.

That is why VAEs sit at the intersection of maximum likelihood, Bayesian inference, and KL divergence.

Two natural next moves branch from this page. Normalizing flows ask what changes when the latent transformation is invertible and likelihood stays exact. Diffusion asks what changes when generation is learned as a gradual denoising process rather than as one sampled latent code.

Study prompts for this section: Intuition

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Mathematics

Definitions, assumptions and the derivation.

Equation 1 Equation 2 Equation 3

Latent variable likelihood

For one observed datapoint $x$ , assume a prior $p(z)$ and a decoder likelihood $p_\theta(x\mid z)$ . The latent-variable model, marginal likelihood, and true posterior are

\begin{aligned} p_\theta(x,z)&=p(z)p_\theta(x\mid z),\\ p_\theta(x)&=\int p_\theta(x,z)\,dz,\\ p_\theta(z\mid x)&=\frac{p_\theta(x,z)}{p_\theta(x)}. \end{aligned}

The evidence $p_\theta(x)$ appears in the denominator, so exact posterior inference and exact likelihood training are tied to the same difficult integral. The whole method rests on three lines:

\begin{aligned} \log p_\theta(x) &= \mathcal L(\theta,\phi;x) + \mathrm{KL}\!\left(q_\phi(z\mid x)\,\|\,p_\theta(z\mid x)\right),\\ \mathcal L(\theta,\phi;x) &= \mathbb E_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)] - \mathrm{KL}\!\left(q_\phi(z\mid x)\,\|\,p(z)\right),\\ \epsilon&\sim\mathcal N(0,I), \qquad z=\mu_\phi(x)+\sigma_\phi(x)\odot\epsilon. \end{aligned}

The first line says the ELBO gap is exactly the KL from the learned encoder $q_\phi(z\mid x)$ to the true posterior. The second line separates reconstruction likelihood from prior pressure. The third line keeps randomness in external noise $\epsilon$ , so gradients of sampled reconstruction terms can flow through $z$ into the encoder outputs.

The ELBO identity

Let $q_\phi(z\mid x)$ be any approximate posterior whose support is compatible with the true posterior. To keep the equations readable, write $q(z)$ for $q_\phi(z\mid x)$ and $\mathbb E_q$ for expectation under that distribution:

\mathbb E_q[\cdot] \equiv \mathbb E_{q(z)}[\cdot].

Using Bayes' rule and rearranging gives the exact identity

\log p_\theta(x)=\mathcal L+G.

Here $\mathcal L$ is the ELBO and $G$ is the inference gap. Let $p_*(z)$ denote the true posterior $p_\theta(z\mid x)$ . Then the gap is

G=\mathrm{KL}(q(z)\,\|\,p_*(z)).

Equivalently,

G=\mathbb E_q[\log q(z)-\log p_*(z)].

The ELBO term is

\mathcal L = \mathbb E_q[\log p_\theta(x,z)-\log q(z)].

The gap can also be read by subtraction:

G=\log p_\theta(x)-\mathcal L.

KL is nonnegative, so

\mathcal L(\theta,\phi;x)\le \log p_\theta(x).

Maximizing the ELBO improves this lower bound. The bound is tight exactly when the encoder distribution equals the true posterior, up to probability-zero events.

Reconstruction likelihood plus prior KL

Because

\log p_\theta(x,z) = \log p_\theta(x\mid z)+\log p(z),

the ELBO can be rewritten as

\begin{aligned} \mathcal L &= \mathbb E_q[\log p_\theta(x\mid z)] \\ &\quad- \mathrm{KL}(q(z)\,\|\,p(z)). \end{aligned}

The first term is a reconstruction log likelihood, not generically a pixel distance. The second term keeps the encoder's posterior close to the prior used for generation. At generation time we sample $z\sim p(z)$ , not $z\sim q_\phi(z\mid x)$ for a training example, so keeping the encoder's distributions close to the prior matters.

Two KL divergences now appear, and they are easy to confuse. $\mathrm{KL}(q\,\|\,p(z))$ , to the prior, sits inside the ELBO; $\mathrm{KL}(q\,\|\,p_*)$ , to the posterior, is the gap. An encoder equal to the prior makes the first zero but usually leaves a gap; an encoder equal to the posterior closes the gap but usually raises the first, and the ELBO still goes up. The ELBO bounds one model's $\log p_\theta(x)$ , so a higher ELBO for one model than for another does not mean a higher likelihood unless their gaps are known. The practice after the demo computes both divergences for a model small enough to solve exactly.

Reconstruction likelihood is not always MSE

The reconstruction term is

\mathbb E_q[\log p_\theta(x\mid z)].

It becomes a familiar loss only after choosing a decoder distribution. With a fixed-variance Gaussian decoder centered at $f_\theta(z)$ , the negative log likelihood is a scaled squared error plus a constant. So MSE appears as a Gaussian negative log likelihood only after that modeling choice.

If $x$ is binary and

p_\theta(x\mid z)=\operatorname{Bernoulli}(\pi_\theta(z)),

then the negative reconstruction log likelihood is binary cross-entropy.

Reparameterization trick

For a diagonal Gaussian encoder, the network outputs $\mu_\phi(x)$ and $\sigma_\phi(x)$ . We sample with external noise:

\epsilon\sim\mathcal N(0,I), \qquad z=\mu_\phi(x)+\sigma_\phi(x)\odot\epsilon.

This keeps the randomness in $\epsilon$ and makes the sample differentiable with respect to the encoder outputs $\mu_\phi(x)$ and $\sigma_\phi(x)$ .

Beta-VAE objective

A beta-VAE (Higgins et al., 2017) changes the training objective to

\begin{aligned} \mathcal L_\beta &= \mathbb E_q[\log p_\theta(x\mid z)] \\ &\quad- \beta\,\mathrm{KL}(q(z)\,\|\,p(z)). \end{aligned}

When $\beta=1$ , this is the standard VAE ELBO. When $\beta\ne 1$ , it is a modified objective rather than the original ELBO. For $\beta>1$ , it remains a lower bound because it subtracts extra nonnegative KL penalty, but it is no longer the standard maximum-likelihood ELBO. For $\beta<1$ , it is generally not guaranteed to lower-bound $\log p_\theta(x)$ .

Larger $\beta$ puts more pressure on $z$ to carry less information and stay close to the prior. This can encourage more factor-like representations in some settings, but it does not guarantee disentanglement: Locatello et al. (2019) show that learning disentangled representations without supervision is impossible without inductive biases on both the model and the data. If the pressure is too strong, the encoder may approach $q_\phi(z\mid x)\approx p(z)$ , so $z$ carries little information about $x$ .

Study prompts for this section: Mathematics

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Code

The same calculation as runnable code, in the notation of the derivation.

Code example 1import math def log_norm(x, mean, var): return -0.5 * (math.log(2 * math.pi * var) + (x - mea...python

This scalar linear-Gaussian example is small enough that we can compute the exact marginal likelihood, exact posterior, ELBO, and inference gap, then check a one-sample reparameterized reconstruction gradient by finite differences.

import math

def log_norm(x, mean, var):
    return -0.5 * (math.log(2 * math.pi * var) + (x - mean) ** 2 / var)

def kl_norm(mu0, var0, mu1, var1):
    return 0.5 * (var0 / var1 + (mu1 - mu0) ** 2 / var1 - 1 + math.log(var1 / var0))

x, w, b, sigma_x = 1.5, 1.2, -0.1, 0.3  # z~N(0,1), x|z~N(wz+b,sigma_x^2)
var_x = sigma_x ** 2
mu_q, logvar_q = 0.3, -0.7              # q(z|x)=N(mu_q, exp(logvar_q))
var_q = math.exp(logvar_q)

log_px = log_norm(x, b, w * w + var_x)
post_var = 1 / (1 + w * w / var_x)
post_mu = post_var * w * (x - b) / var_x
recon = -0.5 * (math.log(2 * math.pi * var_x) + ((x - b - w * mu_q) ** 2 + w * w * var_q) / var_x)
kl_prior = kl_norm(mu_q, var_q, 0, 1)
elbo = recon - kl_prior
gap = log_px - elbo
kl_post = kl_norm(mu_q, var_q, post_mu, post_var)
assert abs(gap - kl_post) < 1e-10

eps = -0.4
def sampled_recon(mu, logvar):
    sigma = math.exp(0.5 * logvar)
    z = mu + sigma * eps
    return log_norm(x, w * z + b, var_x), z, sigma

sample_loglik, z, sigma = sampled_recon(mu_q, logvar_q)
dloglik_dz = (x - (w * z + b)) * w / var_x
path_mu = dloglik_dz
path_logvar = dloglik_dz * 0.5 * sigma * eps
h = 1e-5
fd_mu = (sampled_recon(mu_q + h, logvar_q)[0] - sampled_recon(mu_q - h, logvar_q)[0]) / (2 * h)
fd_logvar = (sampled_recon(mu_q, logvar_q + h)[0] - sampled_recon(mu_q, logvar_q - h)[0]) / (2 * h)
assert abs(path_mu - fd_mu) < 1e-7
assert abs(path_logvar - fd_logvar) < 1e-7

print(round(elbo, 3), round(log_px, 3), round(gap, 3), round(sample_loglik, 3))

For a batched neural VAE, typical shapes are:

x: (B, D)
encoder outputs mu, logvar: (B, K)
noise eps: (S, B, K)
latent samples z = mu[None, :, :] + exp(0.5 * logvar)[None, :, :] * eps: (S, B, K)
decoder log likelihood per sample: (S, B)
Monte Carlo reconstruction estimate: (B,)
analytic diagonal-Gaussian KL: (B,)
per-example ELBO: (B,)
optimization loss -elbo.mean(): scalar

Study prompts for this section: Code

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Demo

Change an input, predict the result, and see which quantities respond.

ExploreChange one control at a time and name what responds.An optional observation guide follows the demo.

Explore Variational Autoencoders

Loading the interactive demo…

Observation guide

Optional and ungraded. Opening a guide saves nothing.

Choose what to inspect in Variational Autoencoders, then open its guide.

The demo keeps the latent variable one-dimensional so the hidden inference geometry can be revealed after a prediction. Move the approximate posterior $q(z\mid x)$ , inspect the prior and likelihood shape, then predict what the ELBO gap will diagnose.

After reveal, compare the target curve and identity values with your prediction. The point is to feel the ELBO as a lower bound whose looseness is measured by the mismatch between $q$ and the hidden target distribution.

Study prompts for this section: Demo

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Practice

In a model whose posterior and evidence are exact, compute the KL to the prior and the KL to the posterior for two encoders. Then compare two models by their ELBOs.

Loading the exercise…

Focus on

In focus

Concept: Variational Autoencoders

What is the smallest example of Variational Autoencoders you can work through by hand, and what does it show?

BeforeMaximum LikelihoodNextNormalizing Flows: Tractable Density via Invertible Transforms

DetailsGenerative Models

About this itemGenerative Models

5 sources citedChecking this browser

Reference id

concept:generative-models/vaes

Math3 equationsp_\theta(x,z)=p(z)p_\theta(x\mid z); p_\theta(x)=\int p_\theta(x,z) dz; p_\theta(z\mid x)=(p_...CodeVariational Autoencoders code example 1assert abs(gap - kl_post) < 1e-10 DemoVariational Autoencoders interactive demoChange one control at a time and name what responds.

After the first pass

Go deeper

Optional material for a second reading. Sources are always shown; the other panels open from their headings or from these links, and closing one keeps what you wrote or revealed.

What to look for in each sectionChoose a section and what you expect to see in it, then read short advice.

Section by section

Choose what to look for in each section

Choose what to look for (no demo yet)01 / Intuition

Illustration of data encoded into a latent distribution and decoded back into reconstructed samples.

What to look for

Start with the picture, metaphor, or geometric mechanism.

Choose first

Choose what you expect to see in this section of Variational Autoencoders.

Questions to ask of the pictureChoose a part of the picture and what you expect, then read short advice.

Looking at the picture

Questions to ask of the picture

4 of 4 sections writtenNo live demo yet

Visual cueWhich part of the picture should you look at first?

Highlight size2/4

Question

Which part of the picture should you look at first?

Choose first

Pick the part of the picture you expect to explain Variational Autoencoders.

Sources

References for this page

A reference shows where an idea comes from; it does not vouch for every step on this page.

paper · 2014Auto-Encoding Variational BayesKingma and Welling

Why it is cited

Introduces the variational autoencoder: a learned recognition model, the evidence lower bound and the reparameterization trick (ICLR 2014).

Open source

paper · 2014Stochastic Backpropagation and Approximate Inference in Deep Generative ModelsRezende, Mohamed, and Wierstra

Why it is cited

Independently develops stochastic backpropagation for latent-variable generative models with learned inference networks (ICML 2014).

Open source

paper · 2017beta-VAE: Learning Basic Visual Concepts with a Constrained Variational FrameworkHiggins et al.

Why it is cited

Weights the KL term by beta > 1 and reports more disentangled latent factors on several image datasets (ICLR 2017).

Open source

paper · 2019Challenging Common Assumptions in the Unsupervised Learning of Disentangled RepresentationsLocatello et al.

Why it is cited

Proves that unsupervised disentanglement is impossible without inductive biases on model and data; in more than 12,000 trained models, well-disentangled ones could not be identified without supervision (ICML 2019).

Open source

paper · 2019An Introduction to Variational AutoencodersKingma and Welling

Why it is cited

A monograph-length treatment of VAEs: the ELBO and its gap, the reparameterization trick and common extensions (Foundations and Trends in Machine Learning, 2019).

Open source

What each claim rests on1 claim, each with its sources, where it appears on this page, and its caveats.

Claims and sources

A latent-variable model trained by maximizing the evidence lower bound (ELBO). The bound falls short of the log-likelihood by the KL divergence from the encoder to the true posterior, so the encoder learns approximate inference, not just compression.

Checked against the cited passages

The checks were made by this site, not by an independent reviewer.

A VAE fits a latent-variable generative model by maximizing the ELBO: q_phi(z|x) approximates the usually intractable posterior, E_q[log p_theta(x|z)] rewards reconstruction likelihood, KL(q_phi(z|x)||p(z)) penalizes deviation from the prior, and reparameterization gives pathwise gradients through z samples.

Sources

Auto-Encoding Variational BayesIntroduces the variational autoencoder: a learned recognition model, the evidence lower bound and the reparameterization trick (ICLR 2014).Stochastic Backpropagation and Approximate Inference in Deep Generative ModelsIndependently develops stochastic backpropagation for latent-variable generative models with learned inference networks (ICML 2014).

What the sources support

Kingma/Welling and Rezende/Mohamed/Wierstra both ground VAEs/deep latent-variable models in ELBO optimization with learned inference networks and reparameterized or stochastic backpropagation estimators. The...

Where to see it on this page

Auto-Encoding Variational Bayes Stochastic Backpropagation and Approximate Inference in Deep Generative Models Equation 1 Equation 2 Code example 1 Demo

Equation 1

\begin{aligned} p_\theta(x,z)&=p(z)p_\theta(x\mid z),\\ p_\theta(x)&=\int p_\theta(x,z)\,dz,\\ p_\theta(z\mid x)&=\frac{p_\theta(x,z)}{p_\theta(x)}. \end{aligned}

Equation 2

\begin{aligned} \log p_\theta(x) &= \mathcal L(\theta,\phi;x) + \mathrm{KL}\!\left(q_\phi(z\mid x)\,\|\,p_\theta(z\mid x)\right),\\ \mathcal L(\theta,\phi;x) &= \mathbb E_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)] - \mathrm{KL}\!\left(q_\phi(z\mid x)\,\|\,p(z)\right),\\ \epsilon&\sim\mathcal N(0,I), \qquad z=\mu_\phi(x)+\sigma_\phi(x)\odot\epsilon. \end{aligned}

Caveat

The reconstruction term is an expectation under q_phi and often Monte Carlo estimated. Pathwise gradients require a differentiable transform of parameter-independent noise, so this does not cover discrete or...

StatusChecked against the cited passages.

Kingma/Welling support q_phi(z|x) as a learned recognition model for the intractable posterior, derive log p_theta(x)=ELBO+KL(q||posterior), decompose the ELBO into E_q log p_theta(x|z)-KL(q||p(z)) and give Gaussian reparameterization. Rezende et al. independently support the recognition-model, lower-bound/free-energy, reconstruction/regularization and stochastic-backprop structure. The page's equations, code and demo show the ELBO gap, prior KL, a reparameterized sample and posterior checks shown once you predict.

Explain it without the pageWrite your own explanation before asking for feedback. Your draft and any hints stay while this page is open.

Practice · Variational Autoencoders

Try the idea in your own words

Concept · Selected for practice

Variational Autoencoders

What it rests on: Sources: Auto-Encoding Variational Bayes; Stochastic Backpropagation and Approximate Inference in Deep Generative Models; beta-VAE: Learning Basic...

Context and links

Generative Models

Choose a task

Explain the mechanism

For Variational Autoencoders: What is the smallest example of Variational Autoencoders you can work through by hand, and what does it show? Explain your answer, including what changes, why, and which assumption matters.

Your answerNo answer yet

A rough first thought is enough. Your draft stays when you change tasks.

Your draft stays on this page and is cleared when you leave or reload.

A little help · Explain

Open one hint at a time. These are suggestions, not your answer or a grade.

0 of 3 hints shown for this question.

Clearing your answer keeps this help record. Outside help is your own report; this page cannot check it.

Where am I stuck? (optional)

Your own description, not an automatic diagnosis

Choose one, or leave this unspecified. Select it again to clear it.

Take your draft to a feedback conversation

No AI feedback runs here. You can copy a prompt to use elsewhere; nothing is sent automatically. Review the text before sharing, and leave out private information.

Write an attempt before copying a feedback prompt.

A draft, or an AI reply to it, is not a test of what you have learned. To check that, try a different case later without help.

Ask about this pagePick one item on this page and copy a prompt about it into an AI assistant you already use. Notes you write here stay in this browser.

Ask about this pageClose

ConceptVariational AutoencodersSources: Auto-Encoding Variational Bayes; Stochastic Backpropagation and Approximate Inference in Deep Generative Models; beta-VAE: Learning Basic...assert abs(gap - kl_post) < 1e-10

Your question

Choose what your question is about

Pick the idea, equation, source, code, claim, misconception or demo state you want to ask about. The prompt you copy includes it, so the answer can stay on that item.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptGenerative Models

Variational Autoencoders

Starting question

What is the smallest example of Variational Autoencoders you can work through by hand, and what does it show?

SourcesCheck the 5 cited sources listed under Sources.Notes can be saved for this item

Points of view on this item

Fixed prompts generated from this item. They are not comments from people or independent reviews.

Learner’s first stepAsk what would make "Variational Autoencoders" feel predictable rather than familiar.

Assumption

The 5 cited sources must support this exact item, not just the surrounding topic.

Source-checking summary

Connect the definition to one equation, piece of code or demo before widening the discussion.

Proposed experiment

Change one input, then check that the same relationship holds in the mathematics, the code and the demo.

Next action

You can state the mechanism in your own words

Evidence4 checks

ObservationChecking for a saved observation

ActionReady for one action

PromptLearner prompt ready to copy

Go to this item

01ObservationChecking what this browser has saved

02EvidenceChecking for a saved observation

03SourcesCheck the 5 cited sources listed under Sources.

04Next stepSave one next action

Local action draftNo local draft saved yetOpen when you are ready to write one next action

Local action draft

This draft stays in this browser and is attached to this item only.

Draft noteNext action

No local draft saved.

Evidence to inspect

What the 5 cited sources say about this exact item
The definition, its prerequisites, and a contrasting concept
The equation or code that makes the concept concrete
One demo state that shows what stays the same as the inputs change

What would resolve this

You can state the mechanism in your own words
You can name the prerequisite that would clear up the confusion
You can predict how the result changes when one input changes

Reference id

concept/concept-notebook/generative-models/vaes
concept:generative-models/vaes

Return to reading

Reading map

In this concept

Change one control at a time and name what responds.

Try the demo

Continue the curriculum

Part IV · unit 1 of 5

Normalizing Flows: Tractable Density via Invertible Transforms

Connections and sources

Domain

Generative Models

generative-models
vae
variational-inference
elbo
kl-divergence

Prerequisites

Leads to

Score Matching & Score-Based Generative Models

Study aid

Ask an assistant about this notebook

Compose a question with this notebook as context, then copy it into an AI assistant you already use.

Compose a question

Your question

GoalComfortStyleStuck on

Copy-only tool: this panel composes a prompt from this page and your choices above, for an AI assistant you already use. Nothing you enter here is sent from this page.