Speculative decoding lowers latency without changing the output distribution: Leviathan et al. (2023) report 2–3× faster generation from T5-XXL with the same output distribution.
A newer notebook covers this concept. This older page keeps its summary, math and sources.
Concept 20 of 100Speculative DecodingEfficiency · Study phase 6: Modern efficiency & inference
Question
Why does accepting a draft token with probability min(1, p/q) and resampling rejections from the normalized max(0, p − q) give samples exactly from the target distribution p?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueContinue in the newer notebook.
The newer notebook covers the same concept and has an interactive demo.
Why It Matters for Modern Models
Speculative decoding lowers latency without changing the output distribution: Leviathan et al. (2023) report 2–3× faster generation from T5-XXL with the same output distribution.
Chen et al. (2023) report a 2–2.5× decoding speedup for the 70B-parameter Chinchilla model in a distributed setup.
Tree-based speculation verifies several candidate continuations in one target pass; Sequoia reports up to about 4× faster decoding for Llama 2 7B on an A100 GPU (Chen et al., 2024).
Per-token latency matters most in interactive use such as chat and code completion.
The verification pass attends over the cached prefix plus the k draft tokens, so the KV-cache and FlashAttention techniques of Efficient Attention apply to it directly.
Questions to Think About
How close must the draft model be to the target, and why does agreement on the current context matter more than overall quality?
Why is checking k draft tokens in one target pass nearly as cheap as generating a single token?
When does exploring a tree of candidate continuations beat a single draft sequence, given the extra computation it spends?
Interactive Demo
Core Math
If you prefer intuition first, try the demo above, then come back for the math.
Key Equation
αi=min(1,qi[xi]pi[xi])
An autoregressive language model produces one token per forward pass of the full model, so generation is sequential. Speculative decoding uses a small, fast draft model to propose k tokens, then checks all of them with one parallel forward pass of the target model.
The method is exact: the output tokens, including residual replacements, are distributed exactly as if they had been sampled from the target model (Leviathan et al., 2023; Chen et al., 2023).
Acceptance probability for each draft token xi:
αi=min(1,qi[xi]pi[xi])
where pi is the target model distribution and qi is the draft distribution at position i.
Residual sampling at the first rejected position:
xi∼Normalize(max(0,pi−qi))
Accepting with probability αi and resampling rejections from this residual gives a token distributed exactly as pi.
Speedup. If each draft token is accepted with probability α (treated as independent), one target pass yields on average
1−α1−αk+1
tokens, and the expected wall-time improvement is (1−α)(kc+1)1−αk+1, where c is the cost of one draft step relative to one target step (Leviathan et al., 2023).
Key Papers
Fast Inference from Transformers via Speculative Decoding
Which limitations does it address, or leave to a later idea?
Latency reduction: a limitation of Speculative Decoding that LLM Serving addresses. Open LLM Serving
Speculative decoding spends extra computation to cut latency: a small model drafts several tokens and the large model checks them all in one parallel pass. It keeps the large model's output distribution exactly and reduces the time per generated token; it does not speed up processing of the prompt (Leviathan et al., 2023).
Autoregressive latency: a limitation of Attention that Speculative Decoding addresses. Open Attention
Token-by-token decoding is the sequential bottleneck in transformer inference. Speculative decoding was invented to propose many tokens cheaply and verify them in parallel without changing the target distribution.
Where does it break down?
Low acceptance: where Speculative Decoding breaks down, taken up in LLM Serving. Open LLM Serving
Speculative decoding speedups rely on high draft-token acceptance. Under distribution shift, long contexts, or mismatched decoding settings, acceptance can drop and the method becomes neutral or even slower.
High entropy lowers acceptance: where Speculative Decoding breaks down, taken up in Decoding & Sampling. Open Decoding & Sampling
Speculative speedups depend on draft tokens being accepted. High-entropy settings, such as a high temperature or a loose top-p, make the draft and target distributions disagree more often, which lowers acceptance and can erase the speedup.
Where else does the same idea appear?
Student as draft: an analogy between Speculative Decoding and Efficiency. Open Efficiency
Distillation trains a smaller student to mimic a large teacher. Speculative decoding uses a small "student-like" draft model to propose tokens, then the teacher verifies—same teacher/student pattern at inference time.
Draft-verify routing: an analogy between Speculative Decoding and MoE. Open MoE
Both put a cheap component in front of an expensive one. An MoE router, a small linear layer, decides which experts run for each token; a small draft model proposes tokens that the large target model then checks. The analogy is loose: routing saves computation by skipping experts, while speculative decoding still runs the target model and saves time by checking several tokens in one pass.
Controlling sampling: the same technique appears in Decoding & Sampling. Open Decoding & Sampling
Both decide which tokens get emitted without retraining the model. Speculative decoding uses accept/reject sampling to reproduce the target model's distribution exactly; top-k and top-p sampling deliberately change the distribution by cutting off its tail.
Hide latency with parallelism: the same technique appears in MoE Serving. Open MoE Serving
MoE serving overlaps the all-to-all communication of routed tokens with computation, for example by pipelining micro-batches (Zhu et al., 2025); speculative decoding checks several drafted tokens in one parallel pass to shorten a sequential chain of steps. Both use parallel work to hide a sequential cost.
You can sample from p directly, or sample from a proposal q and accept/reject to exactly recover p. Speculative decoding is this sampling duality applied to autoregressive generation.