Continuous Function

Study Module

Attention, KV-cache memory and decoding

Move from the attention equation to KV-cache memory, the tradeoffs of grouped-query and multi-query attention (GQA, MQA), how FlashAttention schedules memory, what grows with long context, serving latency, and decoding, all on one page.

interactiveKV-cache calculatorfour equationsdecoding controls
Paper-mapping statusStart with attention
Illustration of a KV cache as a tabletop model: rows of key and value blocks narrowing to fewer shared blocks.
01QK^T
02KV cache
03GQA
04TPOT
KV-cache memoryMem_KVB * N_layers * T * H_kv * d_head * 2 * bytes

Ways in

Choose where to start.

Follow the stages in order, start from a claim, start from the calculator, or keep the equations in view. All four use the same equations and calculator.

01AnswerWhich symbol controls cache size?02EquationKV cache memory03Experimentchange H_kv only04NextAttention

Study Module

From the attention equation to serving.

Seven stages, read as one mechanism: attention decides how much each new token draws from the stored values, the cache decides what is kept for reuse, and serving turns each symbol in the equations into a cost in memory, latency or quality.

weighted average chosen by content

Attention

How much should this query draw from each earlier token?

One weighted average of value vectors per head.Read the concept page

Equations

The four equations this path uses.

Mem_KV = B * N_layers * T * H_kv * d_head * 2 * bytes
Bactive batch sizescalar
N_layersnumber of transformer layersscalar
Tcached tokens per sequencescalar
H_kvkey/value heads after MHA, GQA, or MQA sharingscalar
d_headwidth of each key/value headscalar
2keys plus values are both cachedconstant
bytesbytes per scalar for the chosen precisionscalar
Taught inEfficient Attention / LLM Serving

Double the context and the cache doubles. During decoding the whole cache is read for every generated token.

Read the concept

KV-cache calculator

Change one input and watch the cache size.

This calculator starts at a serving-sized case: 4 sequences of 32,768 tokens each. The KV-cache reference works the same model for one sequence of 4,096 tokens, step by step.

Recall questionA paper changes MHA to GQA or MQA. Which memory symbol should move first?

The equation is already shown above, so this is a recall question rather than a prediction. Answer it to turn on the calculator.

Choose an answer to turn on the calculator.

current KV cache68.7 GB

0% smaller than full MHA under these settings.

MHA / GQA / MQA

Same attention equation, different cache width.

MHA32 KV heads

Each query head has its own KV head.

68.7 GB
GQA8 KV heads

Each KV head is shared by a group of 4 query heads.

17.2 GB
MQA1 KV head

All query heads share one KV head.

2.1 GB

Decoding

Sampling controls change the next-token set.

Predict firstHigh temperature, top-k = 1: what survives?

Six candidate tokens with made-up logits. Temperature reshapes their probabilities, top-k and top-p filter the candidate set, and what remains is renormalized. Predict which control decides the result before looking at the token list.

Choose first; the probe will then set the sliders to the high-temperature, top-k = 1 case.

cache54%
memory29%
latency17%
qualitycut
papercut
noisecut

Discussion

Questions to discuss

Pick one question. It becomes the focus of the question you compose for an assistant below.
claimPaper claim

KV compression claim

Starting question

What exactly is being compressed: heads, tokens, values, precision, or cache pages?

SourcesNo cited source is attached; check it before relying on it.Notes cannot be saved for this item yet
Points of view on this item

Fixed prompts generated from this item. They are not comments from people or independent reviews.

Learner’s first stepAsk what would make "KV compression claim" feel predictable rather than familiar.
Assumption

No source is attached yet; treat claims about sources as provisional.

Source-checking summary

The claim should end up supported by a source, weakened, or left explicitly unverified.

Proposed experiment

Ask what concrete observation would raise or lower confidence in this exact claim.

Next action

The claim is either source-supported, weakened, or marked unverified

Evidence3 checks
ObservationChecking for a saved observation
ActionCannot be saved yet
PromptLearner prompt ready to copy
01ObservationChecking what this browser has saved
02EvidenceChecking for a saved observation
03SourcesNo cited source is attached; check it before relying on it.
04Next stepSave one next action
Local action draftDraft unavailableNotes cannot be saved for this item yet
Local action draft

Notes cannot be saved for this item yet.

No local draft saved.
Evidence to inspect
  • The exact source passage, or the detail in the paper, behind the claim
  • The equation, concept or small experiment that could refute the claim
  • Benchmarks, assumptions, and counterexamples that would change confidence
What would resolve this
  • The claim is either source-supported, weakened, or marked unverified
  • The mechanism is separated from benchmark or marketing language
  • You know what evidence would raise or lower confidence

Ask about one thing

Choose what your question is about.

The question you compose below includes this choice, so it is about one stage, equation, calculator or discussion question rather than the whole page.

EquationKV cache memory

Double the context and the cache doubles. During decoding the whole cache is read for every generated token.

Ready to ask

Study aid

Ask an assistant about this notebook

Compose a question with this notebook as context, then copy it into an AI assistant you already use.

Compose a question

Copy-only tool: this panel composes a prompt from this page and your choices above, for an AI assistant you already use. Nothing you enter here is sent from this page.