Decoding & Sampling: Temperature, Top-p & Inference-Time Control

Let $z_i$ be the logits for token $i \in \mathcal V$ at some step.

Temperature rescales logits

Temperature $\tau > 0$ produces:

• Smaller $\tau$ sharpens (more deterministic).
• Larger $\tau$ flattens (more exploratory).

Nucleus (top-p) truncation deletes the tail, then renormalizes

Let $S_p$ be the smallest set of tokens whose probability mass is at least $p$ (after sorting by probability):

Then you sample from $p'$, the truncated and renormalized distribution.

This is why top-p changes behaviour: it changes the distribution you sample from (Holtzman et al., 2020, §3.1). Top-k sampling is the fixed-size version: it keeps the $k$ most probable tokens whatever their total probability (Fan et al., 2018).

From conditional draws to a terminal record

The optional sequence investigation uses a different, hand-built model with tokens A, B and EOS (end of sequence). In that token order, the logit weights are [2, 1, 1] at the start, [1, 1, 2] after A, and [1, 2, 1] after B. A logit is the natural logarithm of its weight. Only the last ordinary token affects the next row. Apply temperature first and then top-p to that row, with ties in A, B, EOS order. There is no top-k filter.

Let $q_{\tau,p}(x\mid h)$ be this transformed probability of token $x$ given history $h$. Let $L$ be the maximum number of emitted tokens, including EOS. A record ends on its first EOS, or at $L$ tokens if none is EOS. For a record $y=(y_1,\ldots,y_m)$ in this finite event space, its probability is

No history in the product contains an earlier EOS. In particular, a cutoff has no extra EOS factor. EOS drawn on step $L$ is still a sampled end. The exhaustive records sum to one, including cutoff mass. Filtering can make some records impossible; their probability is zero rather than missing.

At $\tau=1,p=1,L=2$, A → EOS has probability $q(A\mid\varnothing)q(\mathrm{EOS}\mid A)=\tfrac12\tfrac12=\tfrac14$. The second factor comes from the after-A row, not the start row. A → A has probability $\tfrac12\tfrac14=\tfrac18$, and ends by cutoff.

• EOS: probability 1/4; ends by EOS; no answer because it is empty.
• A → EOS: probability 1/4; ends by EOS; answer A.
• B → EOS: probability 1/16; ends by EOS; answer B.
• A → A: probability 1/8; length limit; no answer.
• A → B: probability 1/8; length limit; no answer.
• B → A: probability 1/16; length limit; no answer.
• B → B: probability 1/8; length limit; no answer.

Our answer rule accepts only a nonempty EOS-ended record, using its last ordinary token as the answer. At $L=3$, for example, both A → A → EOS and B → A → EOS contribute to answer A. Empty EOS and every cutoff have no answer under this rule. A consumer that accepts truncated text would need a different extraction rule; it cannot reuse these answer probabilities.

Write $q_A,q_B,q_\varnothing$ for the sums of $Q$ over these three answer classes. At the default they are $1/4,1/16,11/16$. For two independent records, stipulate that A is correct, but let a flawed ranker prefer B over A over no answer, breaking ties in favour of the first record. Then

The latter event is “no B, but at least one A.” Changing temperature, top-p or $L$ requires recomputing $Q$ and these answer probabilities. Dropping cutoff records and rescaling would instead condition on not being cut off.

What comes from the source. Holtzman et al. §§3.1 and 3.3 define nucleus sampling and temperature; in §4.1 their generated passages end at an end-of-document token or a 200-token limit. The three-token example, the length convention, the answer rule and the ranker above are this page's own choices, not a reproduction of their experiment or of any provider's API.

import numpy as np

def softmax(z):
    z = z - z.max()
    ez = np.exp(z)
    return ez / ez.sum()

def top_p_filter(p, probs):
    idx = np.argsort(-probs)
    keep = np.zeros_like(probs, dtype=bool)
    cum = 0.0
    for i in idx:
        keep[i] = True
        cum += probs[i]
        if cum >= p:
            break
    q = probs * keep
    return q / q.sum()

rng = np.random.RandomState(0)
logits = np.array([3.0, 2.0, 1.0, 0.2, -0.5])  # 5 tokens

for tau, p in [(1.0, 1.0), (0.7, 0.9), (1.4, 0.9)]:
    probs = softmax(logits / tau)
    probs = top_p_filter(p, probs)
    samples = rng.choice(len(probs), size=20000, p=probs)
    counts = np.bincount(samples, minlength=len(probs)) / len(samples)
    print("tau=", tau, "top_p=", p, "freqs=", np.round(counts, 3))

Enumerate every stopped record

This standard-library Python code lists every possible output directly: each sequence of A and B tokens, with or without a final EOS. It does not sample or call a model, and it runs on your machine, not in the page. max_tokens includes EOS. The returned table keeps outputs whose probability is zero.

from itertools import product
from math import prod

TOKENS = ("A", "B", "EOS")
WEIGHTS = {"start": (2, 1, 1), "A": (1, 1, 2), "B": (1, 2, 1)}

def terminal_law(temperature=1, top_p=1, max_tokens=2):
    if temperature not in (0.5, 1, 2):
        raise ValueError("Unsupported temperature")
    if top_p not in (0.5, 0.75, 1):
        raise ValueError("Unsupported top_p")
    if max_tokens not in (1, 2, 3):
        raise ValueError("Unsupported max_tokens")
    rows = {}
    for context, weights in WEIGHTS.items():
        # exp(log(weight) / temperature) = weight**(1 / temperature).
        raw = [w ** (1 / temperature) for w in weights]
        probabilities = [v / sum(raw) for v in raw]
        order = sorted(range(3), key=lambda i: (-probabilities[i], i))
        kept, mass = [], 0.0
        for i in order:
            kept.append(i)
            mass += probabilities[i]
            if mass >= top_p:
                break
        rows[context] = [probabilities[i] / mass if i in kept else 0.0
                         for i in range(3)]

    events = []
    for length in range(max_tokens):
        events.extend(prefix + ("EOS",)
                      for prefix in product(("A", "B"), repeat=length))
    events.extend(product(("A", "B"), repeat=max_tokens))
    records = []
    answers = {"A": 0.0, "B": 0.0, "none": 0.0}
    for tokens in events:
        factors = [rows["start" if i == 0 else tokens[i - 1]][TOKENS.index(token)]
                   for i, token in enumerate(tokens)]
        probability = prod(factors)
        stop = "EOS" if tokens[-1] == "EOS" else "length-limit"
        answer = tokens[-2] if stop == "EOS" and len(tokens) > 1 else None
        answers[answer or "none"] += probability
        records.append(dict(tokens=tokens, probability=probability,
                            factors=factors, stop=stop, answer=answer))

    # Enumerate both independent draws. The flawed ranker favours B over A.
    rank = {None: 0, "A": 1, "B": 2}
    covered = selected = 0.0
    for first, second in product(records, repeat=2):
        mass = first["probability"] * second["probability"]
        covered += mass if "A" in (first["answer"], second["answer"]) else 0
        winner = second if rank[second["answer"]] > rank[first["answer"]] else first
        selected += mass if winner["answer"] == "A" else 0
    assert abs(sum(r["probability"] for r in records) - 1) < 1e-12
    return dict(records=records, answers=answers,
                coverageA=covered, selectedA=selected)

if __name__ == "__main__":
    for settings in ((1, 1, 2), (1, 0.75, 2), (1, 1, 1), (2, 1, 3)):
        law = terminal_law(*settings)
        print("temperature, top_p, max_tokens:", settings)
        for record in law["records"]:
            print(record)
        print("Answer masses:", law["answers"])
        print("A coverage / selected:", law["coverageA"], law["selectedA"])

The arithmetic is exact; the right settings are not. Holtzman et al. (2020) showed that always choosing the likeliest words makes a language model's text bland and repetitive, while sampling from the whole distribution lets in the unreliable tail. Nucleus sampling was their answer to both, found by measuring text, not derived. The most probable output can be a poor one: for some translation models, exact search finds that the single most probable translation is often the empty one (Stahlberg and Byrne, 2019). Why the most probable text is so often not the best text is still argued. One proposal is that good text is typical rather than maximal, made of words about as surprising as the model expects on average (Meister et al., 2023). Which rule and which temperature suit a given task is still found by trying.