Study Module
Attention to serving, end to end
Move from the attention equation to KV cache memory, GQA/MQA tradeoffs, FlashAttention's memory schedule, long-context pressure, serving latency, and decoding behavior in one connected workspace.
decode memoryMem_KVB * L * T * H_kv * d * 2
Lab OS Proof Slice
Make this route a complete research room, not just a lesson.
The Lab OS thesis is simple: every concept, equation, source caveat, learner witness, contribution issue, and deployable demo should write into one artifact graph.
0/7 route stages ready; next repair is Attention.
4 equations, 4 canon objects, and 7 stages are addressable.
scripts/lab-os/attention-serving-kv-eval-card.mjs writes a reviewable result artifact.
3 KV-head cases compare MHA, GQA, and MQA with held-fixed variables.
kv_cache_gib passes the local invariant check while benchmark caveats stay attached.
Use the Lab OS eval-card issue template to review or extend this candidate.
For this room, the model-facing proof can start as a serving trace or demo card before any training claim.
Only promote once the room carries provenance, evaluation, contribution history, and user-facing caveats.
Attached source boundariesClaims must travel with caveats.
HELMHELM is a benchmark framework, not evidence that this local toy claim is production-certified.Benchmark provenanceThe route carries a provenance flag; it does not detect real train-test overlap.Lost in the MiddleThis anchors a failure slice; it is not a universal law for every model, prompt, or retrieval setup.PagedAttention / OrcaLocal equations estimate pressure; they do not reproduce vLLM or Orca throughput results.candidate passes local witnessFor fixed batch, layer count, context length, head dimension, and precision, KV-cache memory scales linearly with the number of stored key/value heads.
Toy deterministic calculation for one batch, one context length, one layer count, fixed d_head, fixed precision, and three KV-head choices.
Metric: kv_cache_gib · Output: responses/open-research-lab-os/attention-serving-kv-eval-card-result.jsonReview gatereview ready local witness29/29 checks · AI and human review still pending
MQA / GQAPagedAttention / OrcaHELMFirst contribution issue[Lab OS Eval Card] Attention to Serving KV-cache width pressure
The candidate eval now exists. The next useful contribution is review: tighten the scenario, add a small model-backed task config, or prove that the caveat boundary is too weak.
- Names the scenario, metric, learner slice, and source anchors.
- Lists held-fixed variables before reporting memory changes.
- Compares MHA, GQA, and MQA as separate rows.
Route Studio
Pick a lens, then keep the same evidence moving.
The path should work as a learner map, a claim audit, a lab bench, and a lecture spine without splitting into four different products.
Study Module
One paper route from math to production.
This module reads transformer inference as one continuous mechanism: attention defines the copy operation, cache design decides what can be reused, and serving turns every symbol into memory, latency, and quality tradeoffs.
Attention
Which previous tokens should this query copy from?
A weighted value vector per head.Open conceptCarried Equations
Every formula is a route object.
Mem_KV = B * N_layers * T * H_kv * d_head * 2 * bytessourceEfficient Attention / LLM Serving
Double the context and this term doubles. During decode it is touched on every generated token.
Read the conceptKV Memory Lab
Change the serving budget.
Retrieval checkA paper changes MHA to GQA or MQA. Which memory symbol should move first?
This route has already introduced the equation and reference comparison. Use this as informed retrieval practice before touching the controls; it is not a blind prediction or participant evidence.
Choose an answer to enable the calculator. The equation and reference comparison above are intentionally already visible.
current KV cache68.7 GB
0% smaller than full MHA under these settings.
MHA / GQA / MQA
Same attention equation, different cache width.
MHA32 KV heads
68.7 GBEach query head owns its KV head.
GQA32 KV heads
68.7 GBQuery heads share KV within groups.
MQA1 KV head
2.1 GBAll query heads share one KV head.
Decoding Lab
Sampling controls change the next-token set.
Predict firstHigh temperature, top-k = 1: what survives?
The lab reshapes next-token probabilities with temperature, filters the candidate set, then renormalizes what remains. Predict the binding constraint before inspecting the token list.
Choose first; the probe will then set the sliders to the high-temperature, top-k = 1 case.
Research Room
Keep the argument attached to the exact claim.
Pick a route object before asking for help. The selected paper claim, equation, lab, or misconception becomes the saved focus and companion context.KV compression claim
Anchored question
What exactly is being compressed: heads, tokens, values, precision, or cache pages?
Source boundaryLocal route object; verify evidence before treating it as reviewed content.No stable content-object key yet
Role lenses for this object
These are fixed, deterministic perspectives derived from the selected object. They do not represent people, community contributions, or independent review.
Learner evidence requestAsk what would make "KV compression claim" feel predictable rather than familiar.
No external source is attached yet; keep source claims provisional.
Treat this as an evidence object: the claim should be source-supported, weakened, or left explicitly unverified.
Ask what concrete observation would raise or lower confidence in this exact claim.
The claim is either source-supported, weakened, or marked unverified
Local action draftDraft unavailableNeeds a stable content-object key
Local action draft
This object needs a content object key before local action drafts can attach to it.
No local draft saved.
- Exact source quote or local paper clue that motivates the claim
- The equation, concept, or toy lab that could falsify the claim
- Benchmarks, assumptions, and counterexamples that would change confidence
- The claim is either source-supported, weakened, or marked unverified
- The mechanism is separated from benchmark or marketing language
- The learner knows what evidence would raise or lower confidence
Object-attached AI handoff
I am working in Continuous Function's research reading room. Object: claim - KV compression claim Context: Paper claim Anchor id: claim/attention-serving/what-is-compressed Open question: What exactly is being compressed: heads, tokens, values, precision, or cache pages? Evidence to inspect: - Exact source quote or local paper clue that motivates the claim - The equation, concept, or toy lab that could falsify the claim - Benchmarks, assumptions, and counterexamples that would change confidence Deterministic role lenses for this object: - Boundary: fixed perspectives, not people, community contributions, or independent review - Source-checking summary: Treat this as an evidence object: the claim should be source-supported, weakened, or left explicitly unverified. - Proposed experiment: Ask what concrete observation would raise or lower confidence in this exact claim. - Teach/transfer move: Rewrite the claim as a testable invariant or caveat the learner can carry forward. - Assumptions: - No external source is attached yet; keep source claims provisional. - No stable content-object key is attached yet, so local drafts and memory should stay disabled. - A strong claim requires source support plus a mechanism witness; either one alone is not enough. - Benchmark, novelty, and mechanism claims should not be collapsed into one confidence bucket. - Role-lens requests: - Learner: ask for "Ask what would make "KV compression claim" feel predictable rather than familiar." | assumption: No external source is attached yet; keep source claims provisional. | next action: The claim is either source-supported, weakened, or marked unverified - Researcher: ask for "Exact source quote or local paper clue that motivates the claim" | assumption: No stable content-object key is attached yet, so local drafts and memory should stay disabled. | next action: The mechanism is separated from benchmark or marketing language - Experimenter: ask for "Choose one variable or condition to perturb before asking for an explanation." | assumption: A strong claim requires source support plus a mechanism witness; either one alone is not enough. | next action: The learner knows what evidence would raise or lower confidence - Professor: ask for "Find the smallest transferable rule a learner could reuse without the AI." | assumption: Benchmark, novelty, and mechanism claims should not be collapsed into one confidence bucket. | next action: Teach or transfer: Rewrite the claim as a testable invariant or caveat the learner can carry forward. What would resolve this: - The claim is either source-supported, weakened, or marked unverified - The mechanism is separated from benchmark or marketing language - The learner knows what evidence would raise or lower confidence Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action. Current deterministic role lens for this object: - Role lens: Learner - Evidence request: Ask what would make "KV compression claim" feel predictable rather than familiar. - Assumption to keep visible: No external source is attached yet; keep source claims provisional. - Proposed experiment: Ask what concrete observation would raise or lower confidence in this exact claim. - Next action: The claim is either source-supported, weakened, or marked unverified
claim/attention-serving/what-is-compressedAI Focus Object
Ask about one exact object.
The companion prompt follows this selection, so a question can attach to a stage, equation, lab checkpoint, or discussion anchor instead of floating over the whole page.
Double the context and this term doubles. During decode it is touched on every generated token.
Ready to askObject Companion
Ask beside the selected object
Ask about the current paper claim, equation object, lab setting, saved observation, or discussion anchor. In static preview this remains a grounded prompt surface; when the gateway is configured it becomes live assistance.
Context prompt
You are my AI learning companion for Continuous Function. Current context: Study module: Attention -> Efficient Attention -> RoPE -> FlashAttention -> Long Context -> LLM Serving -> Decoding. Learning surface: Attention to serving route. What this page says: Ask about the current paper claim, equation object, lab setting, saved observation, or discussion anchor. In static preview this remains a grounded prompt surface; when the gateway is configured it becomes live assistance. Current section: Active stage: Attention (content-addressed weighted copy) Active equation: KV cache memory: Mem_KV = B * N_layers * T * H_kv * d_head * 2 * bytes Focused object: Equation - KV cache memory Equation: KV cache memory Mem_KV = B * N_layers * T * H_kv * d_head * 2 * bytes Source: Efficient Attention / LLM Serving Symbols: B scalar, N_layers scalar, T scalar, H_kv scalar, d_head scalar, 2 constant, bytes scalar Stress test: Double the context and this term doubles. During decode it is touched on every generated token. KV lab: B=4, T=32,768, layers=32, H_q=32, H_kv=32, d_head=128, precision=2 bytes/scalar Current KV estimate: 68.7 GB; 0% smaller than full MHA under these settings. Route progress: 0/7 stages ready; next repair Attention. Paper evidence status: checking browser-local route memory; no absence was inferred. Saved lab observation status: checking browser-local route memory; no absence was inferred.. Suggested next step: Commit to the KV memory prediction, then change one serving variable at a time.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Help me inspect a paper claim about KV cache compression. Identify which symbol or system bottleneck the claim changes, what remains fixed, and what evidence I should ask for before believing it. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.
Copy-only tool: this panel composes a prompt for an AI tool you already use. Nothing you enter here is sent from this page.