∇Continuous Function

Transformer Systems Lab

Speculative decoding station

Carry the Serving checkpoint into draft-target verification: a faster path only counts if it preserves the target distribution and actually reduces target-model work.

station 08local LabCase v1prediction firsttarget verification
Start SpeculationBack to ServingOpen Route Cockpit
Selected objectDraft-target verification on one served requestq drafts x1...xk; p verifies; residual repair keeps p
Draft lanecheap qproposes k tokens
Target verifierp checks prefixaccepted tokens advance
Repair laneresidual samplerejected position stays target-correct
01Object

served request plus draft-target step

02Gap

speedup is not correctness

03Prediction

choose the useful intervention

04Manipulation

match, k, cost, verify, repair

05Evidence

accepted prefix and target checks

06Invariant

lossless needs repair; speed needs agreement

07Next

falsify the faster-behavior claim

Carry-in contract

A speedup claim has two gates: correctness and target-step savings.

The route already named serving pressure. Speculative decoding adds a sharper test: draft tokens may make generation faster, but only target verification plus residual repair keeps the sampled distribution honest.

Carried objectServing checkpoint: a concrete workload with TTFT, TPOT, and decode support attached.
PredictionWhich intervention makes the served request faster while preserving the target distribution?
EvidenceAccepted prefix, first rejection, residual repair, target passes, and toy speedup.
Next moveFalsify the faster-behavior claim under an evaluation slice.

Operable Station 08

Speculative decoding

Checking local routeNo prediction yet
  1. 01Text to one update
  2. 02Tokens and position
  3. 03Attention routing
  4. 04RoPE phase
  5. 05KV memory
  6. 06Long-context pressure
  7. 07Serving and decoding
  8. 08Speculative decoding
  9. 09Evaluation and falsification
  10. 10Capstone systems claim

Carry-in from Serving

Latency relief is useful only if target verification keeps the model honest.

Serving named the workload pressure. This station asks whether a fast draft model can reduce target-model steps without turning the output into draft-model sampling.

Prior route objectChecking saved Serving history.
Long-context historyChecking saved Long Context history.
KV historyChecking saved KV history.
Station objectroute:ai-lab/transformers/speculative-decoding#speculative-decoding-station

Selected object

One draft-target verification step on the served request

route:ai-lab/transformers/speculative-decoding#speculative-decoding-station

The object is a verification protocol: draft proposals, target probabilities, accepted prefix, residual repair, and target-step cost.

Prediction checkpoint

Which intervention makes this served request faster while preserving the target distribution?

Manipulation instrument

Change agreement, draft length, draft cost, verification, and repair.

The witness is deterministic and local. It shows which condition the route needs before calling speculation a serving win.

Draft-target match
Draft length k
Draft cost
Target verify
Sampling repair

Serving workload carried in

Keep the request visible while testing the acceleration claim.

Choose an intervention before evidence.

Route spine

After speculation, the route asks what the faster behavior proves.

Speculation can reduce target-model passes in the local witness. The next question is not "is it faster?" but "which bounded claim about behavior survives source-grounded evaluation?"

  1. 01Long-context use

    usable tokens before product pressure

    ready
  2. 02Serving and decoding

    TTFT, TPOT, residency, batching

    ready
  3. 03Speculative decoding

    draft-target verification and residual repair

    active

Route memory

Save the correctness boundary, then continue from memory.

The station writes a browser-local checkpoint with the draft-target object, first prediction, accepted-prefix evidence, target-distribution invariant, speedup boundary, and next repair.

Open Study Memory