Launch Route

Attention to Serving

Follow attention math into KV-cache memory and serving tradeoffs: predict which memory term a method changes, check it in a live calculator, and see what evidence backs each claim. Progress saves in this browser, with scope and reviewer notes below the learner path.

saves in this browserKV prediction labcomparison guardrailcontribution notes

Start with Efficient Attention Open KV prediction lab Review comparison guardrail

A physical KV memory lab with attention heads feeding a smaller cache.

01QK^T

02KV cache

03GQA

04TPOT

decode memoryMem_KVB * L * T * H_kv * d * 2

The route, end to end

One continuous path from attention math to real serving tradeoffs.

This module reads transformer inference as one continuous mechanism: attention defines the copy operation, cache design decides what can be reused, and serving turns every symbol into memory, latency, and quality tradeoffs.

content-addressed weighted copy

Attention

Which previous tokens should this query copy from?

A weighted value vector per head.Open concept

Carried Equations

Pick an equation and inspect every symbol.

Mem_KV = B * N_layers * T * H_kv * d_head * 2 * bytes

Bactive batch sizescalar

N_layersnumber of transformer layersscalar

Tcached tokens per sequencescalar

H_kvkey/value heads after MHA, GQA, or MQA sharingscalar

d_headwidth of each key/value headscalar

2keys plus values are both cachedconstant

bytesbytes per scalar for the chosen precisionscalar

sourceEfficient Attention / LLM Serving

Double the context and this term doubles. During decode it is touched on every generated token.

Read the concept

KV Memory Lab

Change the serving budget.

Predict firstA paper changes MHA to GQA or MQA. Which memory symbol should move first?

Hold B, N_layers, T, d_head, and bytes fixed. A model variant shares K/V heads. Commit to the term before touching the controls.

Choose an answer to unlock the calculator.

Context tokens32,768Layers32Batch4Query heads32KV heads32Real GQA usually uses KV-head counts that divide query heads; this slider shows cache-width scaling.Head dim128

estimated KV cache memory68.7 GB

0% smaller than full MHA under these settings.

MHA / GQA / MQA

Same attention equation, different cache width.

MHA32 KV heads

Each query head owns its KV head.

68.7 GB

GQA32 KV heads

Query heads share KV within groups.

68.7 GB

MQA1 KV head

All query heads share one KV head.

2.1 GB

Decoding Lab

Sampling controls change the next-token set.

Predict firstHigh temperature, top-k = 1: what survives?

The lab reshapes next-token probabilities with temperature, filters the candidate set, then renormalizes what remains. Predict the binding constraint before inspecting the token list.

Choose first; the probe will then set the sliders to the high-temperature, top-k = 1 case.

Temperature0.8Top-k4Top-p0.90

cache54%

memory29%

latency17%

qualitycut

papercut

noisecut

Route Scope

Learn the mechanism before treating it as evidence.

This route connects paper claims, equations, browser demos, and local code checks. It does not report comparison scores, run hosted compute, validate production capacity, or replace expert review.

Where You Are

First route: Attention to Serving

A new learner should see one concrete path: map or search a claim, inspect the route graph, open Efficient Attention, predict the KV-cache lever, review the claim boundary, and leave a reproduction or repair note attached to the exact claim, equation, or demo it concerns.

QuestionHow does attention become a serving bottleneck?

Next repairEfficient Attention

Lab stateKV prediction lab live

Attention to ServingCommit the KV-memory prediction before changing controls.

Reveal which symbol moves, then save a checkpoint.

Next: Efficient Attention concept Open Efficient Attention KV prediction checkpoint

01Homenext action 02Paper Mappernext action 03Knowledge Graphnext action 04Searchnext action 05Attention to Servingnow +1 more

Scope labels requiredPreviews and browser-local state are labeled; claims get review before any comparison; reproduction notes stay attached to the exact claim, equation, or demo they describe.

This early version makes no benchmark, hosted-compute, automatic expert review, or live runtime-performance claims.

Reviewer Evidence

What the route can rely on.

Mechanisms checked against papers, mapped implementation references, protocol notes, and small runnable demos stay separated so open gaps remain visible.

spine stages7partial

claim checked4mechanism anchors

mapped context3serving/eval scope

open gaps6kept explicit

Source coverage tracked internally.Use this lower panel to see what reviewers have mapped and which gaps remain.

Comparison Guardrail

Comparison prep note

Comparison checklist.

Before comparing serving systems, pin the model, serving stack, task or scenario, evaluator, metric, hardware/runtime, contamination caveat, raw artifacts, reviewer path, and toy-vs-production boundary. This route does not report live performance, current vLLM behavior, model-order claims, or production validation.

artifactdraft protocol note

runnot run

resultno scores reported

scopeno production evidence

Level 4 boundaryServing evaluation result artifact contract v0

A scored comparison stays unavailable until setup, results, and review evidence are attached.

requiredModel + weights

Name the exact model family, checkpoint or weights, tokenizer, precision, and any quantization.

requiredServing stack

Pin the serving runtime, release or commit, scheduler/cache settings, batch policy, and dependency versions.

requiredEvaluator version

Pin the evaluator or benchmark harness version, config file, prompt template, and command surface.

requiredScenario / task config

Name the scenario, task, dataset split, request shape, input/output lengths, and any limits.

requiredMetric + aggregation

Declare the metric, aggregation rule, quality target, latency statistic, and confidence interval if used.

requiredHardware / runtime

Record accelerator, CPU, memory, interconnect, driver, OS, batch limits, and concurrency policy.

requiredData + contamination caveat

State dataset version, split, filtering, prompt exposure risk, and whether contamination was checked.

requiredToy vs production boundary

Separate the local formula/demo witness from any measured system result, deployment decision, or public system comparison.

future comparison recordno scores reported

6 result sections must be filled before scores can be shown.

Metric groups after review: quality, latency, throughput, resource_memory.

scopeComparison setup only.

Use this checklist when the route is ready to record setup, result artifacts, and reviewer path.

1 setup gap remains before comparison.

Open evaluation pipeline concept

Learning Lab Ladder

Lab ladder: learn first, then reproduce.

This route is ready for source-grounded reading, browser demos, standalone Level 2 scripts, two local KV-cache checks, and a Level 4 future result contract. Evaluation runs and open contribution tasks are planned next artifacts, not finished claims.

Level 0Read concept13 supported

Level 1Browser demo13 supported

Level 2Toy notebook / script5 supported · 8 partial

Level 3Small reproducible experiment3 supported · 10 planned

Level 4Benchmark / evalnone supported · 13 planned

Level 5Open project contributionnone supported · 13 planned

Attention TransformersLevel 2: Toy notebook / script

L0supportedL1supportedL2partially supportedL3plannedL4plannedL5planned

Not ready: The concept page includes a runnable code example, but no standalone route script with pinned inputs, expected output, and failure note exists yet. Package a standalone route script that prints QK scores, softmax rows, and value mixtures with the same symbol names.

Dot ProductLevel 2: Toy notebook / script

L0supportedL1supportedL2partially supportedL3plannedL4plannedL5planned

Not ready: The concept page includes a runnable code example, but no standalone prerequisite script with pinned inputs, expected output, and failure note exists yet. Package a standalone prerequisite script that shows how dot products become attention logits.

Positional EncodingLevel 2: Toy notebook / script

L0supportedL1supportedL2partially supportedL3plannedL4plannedL5planned

Not ready: The concept page includes a runnable code example, but no standalone route script with pinned inputs, expected output, and failure note exists yet. Package a standalone script that prints position vectors and offset dot products.

RopeLevel 2: Toy notebook / script

L0supportedL1supportedL2partially supportedL3plannedL4plannedL5planned

Not ready: The concept page includes a runnable code example, but no standalone route script with pinned inputs, expected output, and failure note exists yet. Package a standalone RoPE script with explicit phase, offset, and caveat outputs.

Efficient AttentionLevel 3: Small reproducible experiment

L0supportedL1supportedL2supportedL3supportedL4plannedL5planned

Ready: small local experiment. A seeded synthetic request trace keeps GQA/MQA formula-memory ratios stable across context-window and batch-size grids while larger context windows increase formula KV-cache estimates under the clipping rule.Synthetic local experiment; inspect the mechanism before broader evaluation.

Flash AttentionLevel 2: Toy notebook / script

L0supportedL1supportedL2supportedL3plannedL4plannedL5planned

Ready: standalone script. A streaming online-softmax merge matches full softmax attention on fixed scores/values while toy scratch accounting avoids materializing the full score matrix.Local code check; inspect the mechanism before broader evaluation.

Grouped Query AttentionLevel 2: Toy notebook / script

L0supportedL1supportedL2supportedL3plannedL4plannedL5planned

Ready: standalone script. For fixed shape and dtype, the GQA KV-cache ratio against MHA is H_kv / H_q, with each group sharing one KV head.Local code check; inspect the mechanism before broader evaluation.

Long ContextLevel 3: Small reproducible experiment

L0supportedL1supportedL2supportedL3supportedL4plannedL5planned

LLM ServingLevel 2: Toy notebook / script

L0supportedL1supportedL2supportedL3plannedL4plannedL5planned

Ready: standalone script. Toy latency accounting decomposes TTFT plus TPOT, KV-cache formula size, page waste, and explicit SLO pass/fail samples.Local code check; inspect the mechanism before broader evaluation.

Inference Kernels Kv Cache Paged AttentionLevel 3: Small reproducible experiment

L0supportedL1supportedL2partially supportedL3supportedL4plannedL5planned

Ready: small local experiment. A pinned synthetic request trace compares contiguous per-request reservation against paged block-table allocation with full-prefix block reuse, while hiding latency, throughput, runtime, hardware, and benchmark claims.Synthetic local experiment; inspect the mechanism before broader evaluation.

Speculative DecodingLevel 2: Toy notebook / script

L0supportedL1supportedL2partially supportedL3plannedL4plannedL5planned

Not ready: The concept page includes a runnable code example, but no standalone route script with pinned inputs, expected output, and failure note exists yet. Package a standalone speculative decoding script with draft/target outputs and acceptance caveats.

Mixture Of ExpertsLevel 2: Toy notebook / script

L0supportedL1supportedL2partially supportedL3plannedL4plannedL5planned

Not ready: The concept page includes a runnable code example, but no standalone route script with pinned inputs, expected output, and failure note exists yet. Package a standalone MoE routing script that reports token dispatch and load-balance assumptions.

MOE ServingLevel 2: Toy notebook / script

L0supportedL1supportedL2partially supportedL3plannedL4plannedL5planned

Not ready: The concept page includes a runnable code example, but no standalone route script with pinned inputs, expected output, and failure note exists yet. Package a standalone MoE serving toy script with dispatch, all-to-all, and caveat outputs.

Levels 4-5 gated/planned

Two Level 3 local CPU formula-only scripts are available: a cache-width sweep and a paged-allocation block-table check over pinned synthetic requests. Level 4 has a contract template only and no scored result; Levels 4-5 remain gated/planned. No hosted compute, benchmark result, shared GPU job, production validation, current runtime claim, hardware guidance, or live serving measurement is implied.

Later levels stay planned until their artifacts and reviews exist.

Contribution Loop

Contribute to a selected route item.

Pick the exact claim, equation, lab, or gap you checked. The public intake packet carries that item, its evidence, the room task, and the next repair so the Attention to Serving route can improve without becoming a feed. This is an interim public preview surface, not a public GitHub issue board yet.

Selected itemKV cache memoryequation

StatusPublic intake packettemplate backed

Repair noteRoute-specific noteDouble the context and this term doubles. During decode it is touched on every generated token.

Confusing jumpMissing prerequisiteReference-scope issueCode witness failedReproduction succeededReproduction failedPropose open task

Confusing jumpA precise confusion note that can become a bridge, prerequisite, or wording repair.

maintainer triage -> route/content fix -> visible resolution note

Prepare this note

Missing prerequisiteA named missing concept, symbol, or shape that can become a prerequisite bridge.

maintainer triage -> prerequisite route patch -> reviewer check

Prepare this note

Reference-scope issueExact wording plus reference span, with safer reference-scoped wording proposed.

reference triage -> claim wording patch -> GPT Pro or maintainer reference check

Prepare this note

Code witness failedExact command, environment, expected output, and failure output for a runnable witness repair.

maintainer triage -> reproduce locally -> code witness patch -> validator run

Prepare this note

Reproduction reportA success, failure, or partial local run note with caveats intact.

maintainer triage -> artifact note -> scope check -> visible resolution

Prepare this note

Propose open taskOne object-attached task with acceptance criteria, needed sources or artifacts, and reviewer path.

maintainer triage -> scope check -> queue or close with explanation

Prepare this note

feedbacktriagefix or explainreviewer checkvisible resolution

Contribution notes stay attached to a selected claim, equation, lab, or page item.

This preview prepares a private repair note for review; it does not publish anything by itself. Broader evidence claims follow the route scope above.

Research Room

Keep the argument attached to the exact claim.

Pick a route item before asking for help. The selected paper claim, equation, lab, or misconception becomes the saved focus and companion context.

claimPaper claim

KV compression claim

Attached question

What exactly is being compressed: heads, tokens, values, precision, or cache pages?

Local action draftDraft unavailableNo stable key yet

Local action draft

This item doesn't have a stable key yet, so local drafts can't attach to it.

Draft noteNext action

No local draft saved.

Evidence to inspect

Exact source quote or local paper clue that motivates the claim
The equation, concept, or toy lab that could falsify the claim
Benchmarks, assumptions, and counterexamples that would change confidence

What would resolve this

The claim is either source-supported, weakened, or marked unverified
The mechanism is separated from benchmark or marketing language
The learner knows what evidence would raise or lower confidence

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: claim - KV compression claim Context: Paper claim Page anchor: recorded for copy. Open question: What exactly is being compressed: heads, tokens, values, precision, or cache pages? Evidence to inspect: - Exact source quote or local paper clue that motivates the claim - The equation, concept, or toy lab that could falsify the claim - Benchmarks, assumptions, and counterexamples that would change confidence What would resolve this: - The claim is either source-supported, weakened, or marked unverified - The mechanism is separated from benchmark or marketing language - The learner knows what evidence would raise or lower confidence Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

AI Focus

Ask about one specific claim, equation, lab, or misconception.

The companion prompt follows this selection, so a question can attach to a stage, equation, lab checkpoint, or discussion anchor instead of floating over the whole page.

EquationKV cache memory

Double the context and this term doubles. During decode it is touched on every generated token.

Ready to ask

AI Companion

Ask beside the selected object

Ask about the current paper claim, equation, lab setting, saved observation, or discussion thread. Without a live AI connection this panel builds a prompt you can copy; with one connected, it answers in place.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Study module: Attention -> Efficient Attention -> RoPE -> FlashAttention -> Long Context -> LLM Serving -> Decoding. Learning surface: Attention to Serving route. What this page says: Ask about the current paper claim, equation, lab setting, saved observation, or discussion thread. Without a live AI connection this panel builds a prompt you can copy; with one connected, it answers in place. Current section: Active stage: Attention (content-addressed weighted copy) Active equation: KV cache memory: Mem_KV = B * N_layers * T * H_kv * d_head * 2 * bytes Focused object: Equation - KV cache memory Equation: KV cache memory Mem_KV = B * N_layers * T * H_kv * d_head * 2 * bytes Source: Efficient Attention / LLM Serving Symbols: B scalar, N_layers scalar, T scalar, H_kv scalar, d_head scalar, 2 constant, bytes scalar Stress test: Double the context and this term doubles. During decode it is touched on every generated token. KV lab: B=4, T=32,768, layers=32, H_q=32, H_kv=32, d_head=128, precision=2 bytes/scalar Current KV estimate: 68.7 GB; 0% smaller than full MHA under these settings. Formula scope: Toy formula-only KV-memory estimate, not a live serving measurement. Excludes allocator behavior, paged-cache metadata, scheduler effects, tensor layout overhead, live throughput/latency, hardware sizing/deployment guidance, and current vLLM runtime evidence. Route evidence: source-scoped in progress; 4 claim-checked stages, 3 reference-mapped stages, 6 explicit gaps. Evaluation protocol: draft protocol note, not run, no scores reported; fields to pin Model + weights, Serving stack, Evaluator version, Scenario / task config, Metric + aggregation, Hardware / runtime, Data + contamination caveat, Toy vs production boundary; comparison record no scores reported. Lab ladder: 13 concepts mapped; Level 0 13 supported / 0 partial / 0 planned; Level 1 13 supported / 0 partial / 0 planned; Level 2 5 supported / 8 partial / 0 planned; Level 3 3 supported / 0 partial / 10 planned; Level 4 0 supported / 0 partial / 13 planned; Level 5 0 supported / 0 partial / 13 planned. Local code checks 5; small local experiments 2. Levels 4-5 gated/planned. Contribution loop: 6 note types (Confusing jump, Missing prerequisite, Reference-scope issue, Code witness failed, Reproduction succeeded, Reproduction failed, Propose open task); preparing a note does not publish it automatically. Route progress: 0/7 stages ready; next repair Attention. Paper evidence carried: none. No saved lab observation yet.. Suggested next step: Commit to the KV memory prediction, then change one serving variable at a time.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Help me inspect a paper claim about KV cache compression. Identify which symbol or system bottleneck the claim changes, what remains fixed, and what evidence I should ask for before believing it. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.