∇Continuous Function

Transformer Systems Lab

Serving and decoding station

Carry the Long Context checkpoint into deployment pressure: a prompt can be usable and still fail as product behavior when prefill, decode, residency, scheduling, or token support breaks.

station 07local LabCase v1prediction firstspeculation handoff
Start ServingBack to Long ContextOpen Route Cockpit
Selected objectLong-context generation workloadW = prompt, output, requests, pages, decode_mode
PrefillTTFTprompt enters once
DecodeTPOTK/V read every step
Schedulerbatchiteration by iteration
01Object

long-context generation workload

02Gap

usable context vs served behavior

03Prediction

name the bottleneck first

04Manipulation

prompt, output, requests, blocks, decode

05Evidence

TTFT, TPOT, residency, support

06Invariant

serving is a coupled system

07Next

speculative verification

Carry-in contract

A context window becomes product behavior only after serving survives.

The route already separated cache bytes from reliable retrieval. Serving and decoding add a new boundary: the same workload must meet latency, residency, scheduling, and token-support constraints.

Carried objectLong Context checkpoint: usable context has source caveats attached.
PredictionWhich constraint decides whether this workload stays inside local SLO?
EvidenceToy ledger for TTFT, TPOT, page waste, request latency, and token support.
Next moveSpeculative decoding: test draft-target verification without changing target distribution.

Operable Station 07

Serving and decoding

Checking local routeNo prediction yet
  1. 01Text to one update
  2. 02Tokens and position
  3. 03Attention routing
  4. 04RoPE phase
  5. 05KV memory
  6. 06Long-context pressure
  7. 07Serving and decoding
  8. 08Speculative decoding
  9. 09Evaluation and falsification
  10. 10Capstone systems claim

Carry-in from Long Context

Usable context becomes a workload only when serving constraints hold.

Long Context separated fitting tokens from using them. This station keeps that boundary and asks what the product pays when the same case is prefilling, decoding, batching, and sampling for real users.

Prior route objectChecking saved Long Context history.
KV historyChecking saved KV history.
BoundaryServing control is not retrieval quality, and decoding control is not cache allocation.
Station objectroute:ai-lab/transformers/serving-decoding#serving-decoding-station

Selected object

One long-context generation workload under latency constraints

route:ai-lab/transformers/serving-decoding#serving-decoding-station

The object is a workload, not a model slogan: prompt tokens, output tokens, concurrent requests, KV page size, and decoding mode move together.

Prediction checkpoint

When this same long-context case is served to multiple users, which pressure becomes the first bottleneck?

Manipulation instrument

Move workload size, page size, and decoding mode; watch different costs answer.

The metrics are deterministic local estimates. They are designed to expose the shape of the tradeoff, not to benchmark a vendor stack.

Prompt tokens
Output tokens
Concurrent requests
KV block size
Decode mode
Choose a pressure before evidence.

Route spine

After serving pressure, the route tests acceleration honestly.

Speculative decoding is the next station because it can reduce target-model work only when draft tokens verify under the target distribution. That makes it a perfect transfer check after basic serving pressure.

  1. 01KV memory

    H_kv and T make cache bytes visible

    ready
  2. 02Long-context use

    Fitting tokens is not reliable retrieval

    ready
  3. 03Serving and decoding

    TTFT, TPOT, residency, batching, token support

    active

Route memory

Save the invariant, then continue from memory.

The station writes a browser-local checkpoint with the workload object, first prediction, evidence ledger, source caveats, and next repair. The route can resume from the Study Memory surface.

Open Study Memory