Classifier-free guidance combines a diffusion model's predictions with and without a condition, such as a text prompt. It is used in published text-to-image systems such as GLIDE and latent diffusion, the basis of Stable Diffusion, where it greatly improves how closely images follow the prompt (Nichol et al., 2022; Rombach et al., 2022).
Concept 42 of 100Classifier-Free GuidanceGenerative Models · Study phase 9: Advanced architectures & generation
Question
Why does guidance with w > 1 extrapolate beyond the conditional distribution, and why can that give images that follow the prompt more closely but look less natural?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Classifier-free guidance combines a diffusion model's predictions with and without a condition, such as a text prompt. It is used in published text-to-image systems such as GLIDE and latent diffusion, the basis of Stable Diffusion, where it greatly improves how closely images follow the prompt (Nichol et al., 2022; Rombach et al., 2022).
The guidance scale is the main user-facing setting for trading prompt fidelity against diversity in text-to-image models.
One network serves as both the conditional and the unconditional model, because the conditioning is dropped at random during training.
Questions to Think About
What happens at the two extremes of the guidance scale: why does too little guidance ignore the prompt, and too much produce oversaturated images with artifacts?
How is raising the guidance scale like lowering the temperature of a language model, and where does the comparison break down?
Core Math
Key Equation
ϵ~θ=ϵθ(∅)+w⋅(ϵθ(c)−ϵθ(∅))
Classifier-free guidance combines the predictions of one diffusion model run with and without its conditioning c, for example a text prompt:
ϵ~θ(xt,c)=ϵθ(xt,∅)+w⋅(ϵθ(xt,c)−ϵθ(xt,∅))
where w is the guidance scale. With w=1 this is the ordinary conditional prediction; with w>1 it extrapolates past it, away from the unconditional prediction. Text-to-image systems typically use values well above 1. (Ho & Salimans write the same rule with w shifted by one.)
One network learns both predictions: during training the conditioning is replaced by a null token ∅ for a random fraction of examples.
In terms of scores. Because ϵθ is a rescaled score, ϵθ≈−σt∇xtlogp, the rule is equivalent to
s~(xt,c)=∇xtlogp(xt)+w⋅∇xtlogp(c∣xt)
using Bayes' rule, ∇logp(xt∣c)−∇logp(xt)=∇logp(c∣xt). Classifier guidance computes the second term with a separately trained classifier on noisy images (Dhariwal & Nichol, 2021); classifier-free guidance obtains it from the difference between the conditional and unconditional predictions. Raising w trades sample diversity for fidelity to the condition.
Key Papers
Classifier-Free Diffusion Guidance
Ho & Salimans2021NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications