Decoding & Sampling: Temperature, Top-p & Inference-Time Control
Decoding turns a model's next-token distribution into a token sequence. Settings such as temperature, top-p and top-k change how each token is drawn, and so whether the same trained model produces repetitive or varied text.
A newer notebook covers this concept. This older page keeps its summary, math and sources.
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueContinue in the newer notebook.
The newer notebook covers the same concept and has an interactive demo.
Why It Matters for Modern Models
Decoding turns a model's next-token distribution into a token sequence. Settings such as temperature, top-p and top-k change how each token is drawn, and so whether the same trained model produces repetitive or varied text.
Decoding controls the balance between determinism and diversity and affects failure modes such as repetition: Holtzman et al. (2020) found that maximization-based decoding such as beam search produces bland, repetitive text, which nucleus sampling reduces.
Changing the decoding procedure needs no training: behavior can be adjusted without touching the weights.
Sampling choices affect reliability: a low temperature reduces variation between runs, but does not prevent the most probable answer from being wrong.
LLM sampling and diffusion guidance both expose a setting that trades diversity for fidelity to the prompt or condition.
Questions to Think About
Why is "temperature controls creativity" an imprecise description, and what does temperature actually do to the softmax?
How does top-p adapt to the entropy of each step's distribution, and why does that make it behave differently from a fixed top-k?
How much of the repetition problem comes from the decoding procedure rather than from training, and which decoding settings reduce it?
In diffusion models, why does very strong guidance degrade images instead of making them ever more faithful to the prompt?
Interactive Demo
Core Math
If you prefer intuition first, try the demo above, then come back for the math.
Key Equation
p′(i)=∑j∈Spp(j)p(i)1[i∈Sp]
Training gives a next-token distribution pθ(⋅∣x<t). Decoding is the step that turns these probabilities into text.
1) Temperature reshapes the softmax
Given logits zi for tokens i∈V, temperature τ gives:
pτ(i∣x<t)=∑jezj/τezi/τ
Lower τ sharpens the distribution and higher τ flattens it. As τ→0 sampling approaches greedy decoding, which always picks the most likely token.
2) Top-k and nucleus (top-p) truncation
Top-k sampling keeps only the k most likely tokens (Fan et al., 2018). Nucleus sampling keeps a variable number: let Sp be the smallest set of tokens, taken in order of decreasing probability, whose total probability is at least p (Holtzman et al., 2020):
Sp=min{S:i∈S∑p(i)≥p}
Then sample from the truncated, renormalized distribution:
p′(i)=∑j∈Spp(j)p(i)1[i∈Sp]
Truncation therefore changes the distribution being sampled from, not just the surface form of the output. Locally typical sampling instead keeps tokens whose information content −logp(i) is close to the entropy of the distribution (Meister et al., 2023).
3) The same idea in diffusion: guidance
Classifier-free guidance in diffusion models also reshapes a distribution at sampling time, with a scale w that trades diversity for fidelity to the conditioning (Ho & Salimans, 2021):
ϵguided=ϵuncond+w(ϵcond−ϵuncond)
In both cases the weights stay fixed; only the sampling procedure changes.
Which limitations does it address, or leave to a later idea?
Fix text degeneration: a limitation of Maximum Likelihood that Decoding & Sampling addresses. Open Maximum Likelihood
Greedy decoding from a model trained by maximum likelihood often falls into repetitive loops. Nucleus sampling was introduced as a decoding-time fix that avoids those loops while cutting off the unreliable low-probability tail, without retraining (Holtzman et al., 2020).
Where does it break down?
Repetition loops: where Decoding & Sampling breaks down, taken up in Maximum Likelihood. Open Maximum Likelihood
Greedy or beam-search decoding from a model trained by maximum likelihood often falls into repetitive loops (Holtzman et al., 2020). Sampling with temperature, top-k or top-p adds randomness that avoids them, at the cost of no longer choosing the single most likely continuation.
High entropy lowers acceptance: where Speculative Decoding breaks down, taken up on this page. Open Speculative Decoding
Speculative speedups depend on draft tokens being accepted. High-entropy settings, such as a high temperature or a loose top-p, make the draft and target distributions disagree more often, which lowers acceptance and can erase the speedup.
Sampling the tail: where RLHF breaks down, taken up on this page. Open RLHF
RLHF shapes behaviour mostly on the kinds of output seen during feedback. Sampling at high temperature or with loose truncation reaches outputs that feedback rarely covered, so aligned behaviour there is less well tested.
Where else does the same idea appear?
Inference-time control: an analogy between Decoding & Sampling and Activation Steering. Open Activation Steering
Decoding changes behavior by reshaping the output distribution (logits → sampling); steering changes it by editing hidden states. Both control a model at inference time without retraining, and both degrade the output when pushed too far: a very high temperature gives incoherent text, and a very strong steering vector moves hidden states into regions the model never saw in training.
Both decide which tokens get emitted without retraining the model. Speculative decoding uses accept/reject sampling to reproduce the target model's distribution exactly; top-k and top-p sampling deliberately change the distribution by cutting off its tail.
Softmax selection: the same technique appears in Attention. Open Attention
Attention uses softmax to select "which positions to copy from," while decoding uses softmax to select "which token to emit." Same pattern: logits → distribution → sample/argmax/weighted sum.
Temperature scaling: the same technique appears in Embeddings. Open Embeddings
Contrastive learning uses temperature τ to control softmax sharpness; decoding temperature does the same to token probabilities. In both places, temperature sets the entropy of a softmax: low values sharpen it, high values flatten it.
Guidance and temperature: an analogy between Decoding & Sampling and Diffusion. Open Diffusion
Classifier-free guidance reweights diffusion trajectories toward conditioning, trading diversity for fidelity. Temperature and top-p reshape token probabilities in a similar way at sampling time. Both are settings applied at inference that move probability mass without retraining.
Top-k then renormalize: the same technique appears in MoE. Open MoE
MoE keeps top-k experts and renormalizes gate. Top-k decoding applies same truncation-and-renormalization to vocabulary logits before sampling.
Preference-aware generation: the same technique appears in KTO. Open KTO
KTO training shapes the model distribution toward human preferences. At inference, sampling strategies (temperature, top-p) further control this shaped distribution—both operate on the same output space with complementary goals.
You can sample from p directly, or sample from a proposal q and accept/reject to exactly recover p. Speculative decoding is this sampling duality applied to autoregressive generation.