A variational autoencoder fits a latent-variable model by balancing reconstruction accuracy against a KL penalty that keeps the encoder’s distribution close to a prior. Stable Diffusion is a latent diffusion model: an autoencoder close to a VAE (it has a small KL penalty on its latents) compresses images, and the diffusion model works in that latent space (Rombach et al., 2022).
A newer notebook covers this concept. This older page keeps its summary, math and sources.
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueContinue in the newer notebook.
The newer notebook covers the same concept and has an interactive demo.
Why It Matters for Modern Models
A variational autoencoder fits a latent-variable model by balancing reconstruction accuracy against a KL penalty that keeps the encoder’s distribution close to a prior. Stable Diffusion is a latent diffusion model: an autoencoder close to a VAE (it has a small KL penalty on its latents) compresses images, and the diffusion model works in that latent space (Rombach et al., 2022).
Autoencoders with VAE-style or vector-quantized latents (VQ-VAE, van den Oord et al., 2017) compress images, audio and video into latents that later generative models operate on.
Questions to Think About
How do the choice of prior p(z) and of the family of approximate posteriors affect sample quality and diversity?
Interactive Demo
Core Math
If you prefer intuition first, try the demo above, then come back for the math.
Key Equation
logpθ(x)≥Eqϕ[logpθ(x∣z)]−KL(qϕ∥p(z))
A latent-variable model pθ(x,z)=p(z)pθ(x∣z) has an intractable posterior pθ(z∣x). A variational encoder qϕ(z∣x) approximates it, and training maximizes the evidence lower bound (ELBO):
The first term rewards accurate reconstruction; the second keeps the encoder's distribution close to the prior. The gap in the inequality is KL(qϕ(z∣x)∥pθ(z∣x)).
The reparameterization trick writes a sample from a Gaussian encoder as a differentiable function of the encoder's outputs, so gradients pass through the sampling step:
Which limitations does it address, or leave to a later idea?
Blurry samples: a limitation of VAEs that Diffusion addresses. Open Diffusion
Samples from VAEs with a simple Gaussian decoder tend to be blurry, because the decoder predicts an average over the images that fit a latent code. Diffusion models were not designed as a fix for VAEs, but they avoid this weakness: read as deep hierarchical VAEs with a fixed noising encoder, they refine an image over many small denoising steps and give much sharper samples (Ho et al., 2020).
Where else does the same idea appear?
Reparameterized noise: the same technique appears in Diffusion. Open Diffusion
VAEs rewrite sampling as z = μ + σ⊙ε so that gradients can pass through the randomness; diffusion writes the noised sample as x_t = √ᾱ_t x_0 + √(1−ᾱ_t) ε. Both push the randomness into a fixed noise variable ε so that gradients can flow.
VAE ELBO contains KL(q(z|x) || p(z))—same KL divergence from MLE, but now regularizing latents toward prior.
Which ideas are its mathematical duals?
Shared ELBO: Diffusion is its mathematical dual. Open Diffusion
Both are trained with a lower bound on log-likelihood, the ELBO. A diffusion model can be read as a hierarchical VAE with one latent layer per noise level and a fixed encoder, the noising process; in the continuous-time limit it has infinitely many layers (Kingma et al., 2021).
Implicit vs explicit density: GANs is its mathematical dual. Open GANs
GANs define a density only implicitly: they produce samples but give no value of p(x). VAEs define an explicit density and are trained on a lower bound on log p(x), which can be estimated but not computed exactly. They are two approaches to the same generative goal.