A diffusion model learns to reverse a process that gradually adds noise to data. Stable Diffusion is a latent diffusion model: a DDPM-style denoiser trained in the latent space of a pretrained autoencoder (Rombach et al., 2022).
A newer notebook covers this concept. This older page keeps its summary, math and sources.
Concept 9 of 100DiffusionGenerative Models · Study phase 4: Generative modeling families
Question
Why is learning to denoise the same as learning the score ∇ₓ log pₜ(x), and in what sense does reverse-time sampling climb toward higher log-density while the injected noise keeps samples spread out?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueContinue in the newer notebook.
The newer notebook covers the same concept and has an interactive demo.
Why It Matters for Modern Models
A diffusion model learns to reverse a process that gradually adds noise to data. Stable Diffusion is a latent diffusion model: a DDPM-style denoiser trained in the latent space of a pretrained autoencoder (Rombach et al., 2022).
OpenAI describes its Sora video model as a diffusion transformer over spacetime patches of compressed video (Brooks et al., 2024).
Straighter sampling paths, as in rectified flow, allow generation in a few steps, and in one step after further distillation (Liu et al., 2023).
Questions to Think About
How do different probability paths, such as the diffusion path and the straight conditional path of flow matching, change the trajectories a sampler follows?
Interactive Demos
Core Math
If you prefer intuition first, try the demos above, then come back for the math.
Key Equation
L=Ex0,t,ϵϵ−ϵθ(xt,t)2
Forward diffusion adds noise:
q(xt∣x0)=N(αˉtx0,(1−αˉt)I)
The model learns to predict the added noise ϵ with a mean-squared error, where xt=αˉtx0+1−αˉtϵ:
L=Ex0,t,ϵϵ−ϵθ(xt,t)2
Ho et al. (2020) obtained this loss by dropping the weights of the variational bound on log-likelihood, so it is a reweighted form of that bound.
Score-based SDE view: the forward process is an SDE dxt=f(xt,t)dt+g(t)dWt. Sampling runs the reverse-time SDE, which needs the score∇xlogpt(x); the noise predictor gives it as ∇xlogpt(xt)≈−ϵθ(xt,t)/1−αˉt.
Flow matching: regress a vector field vθ(x,t) onto the velocity of a chosen conditional path, often the straight line from a noise sample to a data sample (Lipman et al. call this the conditional optimal-transport path). The learned field averages these conditional velocities; it is not in general the optimal-transport map between noise and data.
Key Papers
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Which limitations does it address, or leave to a later idea?
Accelerate sampling: a limitation of Diffusion that Efficiency addresses. Open Efficiency
High-quality diffusion sampling needs tens to hundreds of denoising steps. Distillation methods cut that number: progressive distillation halves the number of steps repeatedly (Salimans & Ho, 2022), and consistency models generate in one or a few steps (Song et al., 2023).
Blurry samples: a limitation of VAEs that Diffusion addresses. Open VAEs
Samples from VAEs with a simple Gaussian decoder tend to be blurry, because the decoder predicts an average over the images that fit a latent code. Diffusion models were not designed as a fix for VAEs, but they avoid this weakness: read as deep hierarchical VAEs with a fixed noising encoder, they refine an image over many small denoising steps and give much sharper samples (Ho et al., 2020).
Fix mode collapse: a limitation of GANs that Diffusion addresses. Open GANs
GAN training is a min-max game that can be unstable and can drop modes of the data. Diffusion models replace it with a regression loss; Dhariwal and Nichol (2021) found that they surpassed the best GANs on image quality while covering more of the data distribution, at the cost of many network evaluations per sample.
Where else does the same idea appear?
Iterative → Recurrent: an analogy between Diffusion and State Space Models. Open State Space Models
Both apply one learned update many times: a diffusion model applies its denoiser once per noise level, and a state-space model applies its state update once per token. The analogy is loose: diffusion steps refine a whole sample, while SSM steps move along the sequence.
Guidance and temperature: an analogy between Diffusion and Decoding & Sampling. Open Decoding & Sampling
Classifier-free guidance reweights diffusion trajectories toward conditioning, trading diversity for fidelity. Temperature and top-p reshape token probabilities in a similar way at sampling time. Both are settings applied at inference that move probability mass without retraining.
Cross-attention: the same technique appears in Attention. Open Attention
Text-to-image diffusion (Stable Diffusion) uses cross-attention to condition on text embeddings—same Q·K^T softmax mechanism, but K,V come from text while Q comes from image latents.
Reparameterized noise: the same technique appears in VAEs. Open VAEs
VAEs rewrite sampling as z = μ + σ⊙ε so that gradients can pass through the randomness; diffusion writes the noised sample as x_t = √ᾱ_t x_0 + √(1−ᾱ_t) ε. Both push the randomness into a fixed noise variable ε so that gradients can flow.
Text-conditional generation: the same technique appears in Multimodal Pretraining. Open Multimodal Pretraining
Text-to-image diffusion models such as Stable Diffusion condition the denoiser on text embeddings through cross-attention (Rombach et al., 2022), so the text guides every denoising step.
With the right weight on each noise level, the noise-prediction loss equals the variational bound on −log p(x) under the fixed noising process, up to constants and a final decoding term; the unweighted "simple" loss used in practice is a reweighted version of that bound (Ho et al., 2020). It is denoising on the surface and approximate likelihood training underneath.
Shared ELBO: VAEs is its mathematical dual. Open VAEs
Both are trained with a lower bound on log-likelihood, the ELBO. A diffusion model can be read as a hierarchical VAE with one latent layer per noise level and a fixed encoder, the noising process; in the continuous-time limit it has infinitely many layers (Kingma et al., 2021).
Adversarial ↔ Denoising: GANs is its mathematical dual. Open GANs
Both turn noise into data. A GAN does it in one pass of a generator trained against a discriminator; a diffusion model does it over many denoising steps, with a denoiser trained by plain regression. Trading the adversarial game for regression makes training more stable, at the cost of slower sampling.