Efficient Attention at Scale: KV Cache, GQA & FlashAttention

How attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.

published · difficulty 4/5 · 26 min read

Intuition

Build the mental picture first so the rest of the page has something to attach to.

PredictName the object in plain language, then predict what should change.Leave with one reusable mental picture before notation appears.

The phrase "attention is $O(T^2)$ " is useful, but it hides which object is causing trouble. Training, prompt processing, and token-by-token decoding do not stress the system in the same way.

During decoding, the model generates one new token at a time. For that new query, it must still compare against all earlier keys and read all earlier values. Recomputing keys and values for the whole prefix would be wasteful, so serving systems keep a KV cache: the keys and values already produced by every layer for every cached token.

That shifts the learner's question from "is attention quadratic?" to a more inspectable object:

If the model keeps the same context length and same head dimension, which symbol changes when full multi-head attention becomes grouped-query attention?

The answer is not the number of query heads. It is the number of stored key/value heads, $H_{kv}$ .

Three mechanisms often get blurred together:

KV caching avoids recomputing past keys and values during autoregressive decoding.
Multi-query/grouped-query attention (MQA/GQA) reduces how many key/value heads are stored and read.
FlashAttention keeps exact attention but changes memory movement during attention computation, especially by avoiding materializing the full attention matrix in high-bandwidth memory.

The first two are about the cached decoding object. FlashAttention is an IO-aware kernel story. They interact in systems, but they are not the same lever.

Section prompt

Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

InspectTrack the same object through the notation and check each symbol.Leave with the invariant the equations preserve.

Equation 1Inspect equation objecto_t = softmax (\frac{q_t K_{1:t}^{^T}}{sqrt(d_k)}) V_{1:t},Equation 2Inspect equation objectMem_{KV} \approx B\cdot N_{layers}\cdot T\cdot H_{kv}\cdot d_{head}\cdot 2 \cdot bytes.Equation 3Inspect equation objecto_h = softmax (\frac{Q_h K_{g(h)}^{^T}}{sqrt(d_k)}) V_{g(h)}.

With a KV cache, the attention output for the new token at time $t$ is:

o_t = \mathrm{softmax}\!\left(\frac{q_t K_{1:t}^{\top}}{\sqrt{d_k}}\right) V_{1:t},

where $(K_{1:t},V_{1:t})$ are cached.

This equation says what the new token reads. The next equation says what the server must keep resident so that read is possible.

KV Cache Memory Scaling

Across a batch of $B$ sequences and $N_{\mathrm{layers}}$ layers, the KV cache stores keys and values for all $T$ positions:

\mathrm{Mem}_{KV} \approx B\cdot N_{\mathrm{layers}}\cdot T\cdot H_{kv}\cdot d_{\mathrm{head}}\cdot 2 \cdot \mathrm{bytes}.

Each symbol is a real system object:

$B$ : active sequences in the batch.
$N_{\mathrm{layers}}$ : transformer layers that each keep their own cache.
$T$ : cached tokens per sequence.
$H_{kv}$ : stored key/value heads per layer.
$d_{\mathrm{head}}$ : width of each head.
$2$ : one tensor for $K$ , one tensor for $V$ .
$\mathrm{bytes}$ : bytes per stored scalar, such as 2 for fp16/bf16.

This is a formula estimate. It does not include allocator fragmentation, paging metadata, scheduler behavior, prefix sharing, tensor layout padding, or model-quality effects from changing the attention architecture.

Grouped-Query Attention

Let there be $H_q$ query heads but only $H_{kv}$ key/value heads, with a mapping $g(h)$ from query head $h$ to a KV head:

o_h = \mathrm{softmax}\!\left(\frac{Q_h K_{g(h)}^{\top}}{\sqrt{d_k}}\right) V_{g(h)}.

If full multi-head attention has $H_q = H_{kv}=32$ and GQA keeps $H_q=32$ but uses $H_{kv}=8$ , then the cache-width term is divided by 4:

\frac{\mathrm{Mem}_{KV}^{\mathrm{GQA}}}{\mathrm{Mem}_{KV}^{\mathrm{MHA}}} \approx \frac{H_{kv}^{\mathrm{GQA}}}{H_{kv}^{\mathrm{MHA}}} = \frac{8}{32} = \frac{1}{4}.

The prediction-first invariant is:

Holding $B$ , $N_{\mathrm{layers}}$ , $T$ , $d_{\mathrm{head}}$ , and bytes fixed, KV-cache memory scales linearly with $H_{kv}$ .

The long-context repair is:

After GQA narrows $H_{kv}$ , increasing $T$ still grows the cache linearly.

For the dedicated head-sharing invariant, selected Q-to-KV mapping, and cache-ratio prediction exercise, see Grouped-Query Attention.

Source Boundaries

Shazeer's multi-query attention work is evidence for the decode-bandwidth motivation: repeated key/value reads can dominate incremental decoding, and sharing K/V heads attacks that object. Ainslie et al. place GQA between full MHA and MQA, where $H_{kv}$ is an architectural choice rather than an afterthought. Dao et al.'s FlashAttention is evidence for a different but adjacent claim: exact attention can become faster when the implementation reduces memory traffic.

Do not use this page to claim that GQA always preserves quality, that FlashAttention reduces the KV cache, or that the formula exactly predicts serving capacity. Those claims need their own evidence.

Section prompt

Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

TraceMatch variables to symbols before reading the implementation.Leave with a runnable witness for the math.

Code witness 1def kv_cache_gb( batch=1, layers=32, tokens=4096, kv_heads=32, head_dim=128, bytes_per_value=...python

def kv_cache_gb(
    batch=1,
    layers=32,
    tokens=4096,
    kv_heads=32,
    head_dim=128,
    bytes_per_value=2,
):
    scalars = batch * layers * tokens * kv_heads * head_dim * 2
    return scalars * bytes_per_value / 1e9

baseline = dict(batch=1, layers=32, tokens=4096,
                head_dim=128, bytes_per_value=2)

full_mha = kv_cache_gb(**baseline, kv_heads=32)
for label, kv_heads, tokens in [
    ("MHA same context", 32, 4096),
    ("GQA same context", 8, 4096),
    ("GQA long context", 8, 32768),
]:
    gb = kv_cache_gb(**{**baseline, "tokens": tokens}, kv_heads=kv_heads)
    print(f"{label:17s} H_kv={kv_heads:2d} T={tokens:6d} "
          f"KV={gb:6.2f} GB  baseline_mha/this={full_mha/gb:4.1f}x")

Section prompt

Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

ManipulateChange one control and predict the visible response before reveal.Leave with the observed invariant or a repaired model.

Live Concept Demo

Explore Efficient Attention at Scale: KV Cache, GQA & FlashAttention

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned

Demo inquiry checkpoint

Manipulate one control and predict the visible change.

01Choose lensTrace a quantity

02ObserveDemo state pending

03GroundName the equation, invariant, or control that explains it.

04CarryNext: LLM Serving at Scale: Prefill, Decode & Continuous Batching

Choose what to inspect in Efficient Attention at Scale: KV Cache, GQA & FlashAttention. This shared fallback is an observation guide, not evidence of learning.

Loading interactive demo...

Use the workbench as a prediction loop, not a calculator:

Select the memory equation as the object.
Predict what changes when $H_q=32$ query heads move from full MHA to GQA with $H_{kv}=8$ .
Reveal same-context evidence.
Increase $T$ and watch the memory pressure return.
Save the invariant: for fixed model shape and precision, $\mathrm{Mem}_{KV}$ scales with $T\cdot H_{kv}$ .

The route handoff is deliberate:

Go to Long Context when the next question is about increasing $T$ , position behavior, and retrieval quality.
Go to LLM Serving when the next question is allocator pressure, batching, latency, and throughput.
Go to FlashAttention when the next question is IO-aware exact attention rather than cache storage.

Section prompt

Copy-only prompts — each action copies a page-grounded prompt to your clipboard. Nothing is sent by this site.

Object

4/4 sections ready

Equation: Efficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2

For Equation 2 in Efficient Attention at Scale: KV Cache, GQA & FlashAttention, what does each symbol mean, what assumption makes it valid, and which source or code witness supp...

BeforeScaled Dot-Product Attention & Transformer LayersNow4/4 sections readyTryManipulate one control and predict the visible change.NextLLM Serving at Scale: Prefill, Decode & Continuous Batching

Object contextExact equation object

StateExact equation object

2 sources attachedCF editorial source-scope reviewChecking local snapshot

equation:attention-transformers/efficient-attention#math-object-2

Math3 equation objectso_t = softmax (\frac{q_t K_{1:t}^{^T}}{sqrt(d_k)}) V_{1:t},CodeEfficient Attention at Scale: KV Cache, GQA & FlashAttention code wit...def kv_cache_gb( batch=1, layers=32, tokens=4096, kv_heads=32, head_dim=128, bytes_per_value=...DemoEfficient Attention at Scale: KV Cache, GQA & FlashAttention predictionManipulate one control and predict the visible change.

EquationLearner lens

Efficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2

For Equation 2 in Efficient Attention at Scale: KV Cache, GQA & FlashAttention, what does each symbol mean, what assumption makes it valid, and which source or code witness supp...

Mode questionCan I say the mechanism back in one sentence before I reveal anything?

Start with the prediction checkpoint, then compare the reveal to the mental model.

Take this move

Study modes

Keep the object fixed; change the lens.

01PredictCommit before reveal.02ExplainName the invariant.03Ground2 source links nearby.04ExperimentChange one knob.05CarrySave the next question.

Efficient Attention Workbench

Make the KV-cache equation behave like an object.

Keep the equation fixed, choose the role lens, predict what grouping changes, then reveal the memory witness.

Selected equationM_KV = B * L * T * 2 * H_kv * d * sLearner lens

1Question

What symbol changed, and what stayed fixed?

Learner

2Object

KV-cache memory equation

M_KV = B * L * T * 2 * H_kv * d * s

3Prediction

Commit a memory prediction.

No prediction committed yet.

4Manipulation

T = 32k, g = 4, s = 2 bytes

H_kv = 8

5Evidence

Hidden until prediction.

8 stored K/V heads

6Invariant

KV-cache memory scales linearly with stored K/V heads.

Query heads stay visible.

7Next

Move g from 1 to 4, then say why the ratio changes before reading the invariant.

Carry this into long context and FlashAttention.

Prediction Gate

If 32 query heads share K/V heads in groups of 4, what happens to KV-cache memory compared with ordinary multi-head attention?

Invariant To CarryFor fixed B, L, T, d, and s, KV-cache memory scales linearly with stored K/V heads, not query heads.

Commit a prediction to reveal the formula-estimated memory.

Open full KV demo Continue in the KV-memory Lab Ask research room

Learner lens

What symbol changed, and what stayed fixed?Move g from 1 to 4, then say why the ratio changes before reading the invariant.

M_KV = B * L * T * 2 * H_kv * d * sH_kv = H_q / g

B = 1L = 32H_q = 32d = 128s = 2 bytes

Sequence length T

KV sharing group size g

Value precision s

32 query heads stay visible

8 stored K/V heads after grouping

8KV headshiddencache at 32khiddenreduction vs MHAhiddenKV-head share vs MHA

Formula-only output — no model, allocator, scheduler, bandwidth, latency, or quality measurement ran.

Pinned evidence for M_KVGQA changes H_kv, not H_q.

Use the cache equation and live lane count to separate memory pressure from query-head count.

Observation ledger

Hidden until the prediction is committed.

Next moveMove g from 1 to 4, then say why the ratio changes before reading the invariant.

KV Cache Object Room

Watch which term changes before comparing efficiency methods.

Local distilled room

Object - EquationEfficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2Question

For Equation 2 in Efficient Attention at Scale: KV Cache, GQA & FlashAttention, what does each symbol mean, what assumption makes it valid, and which source or code witness supports it?

Exact equation objectequation:attention-transformers/efficient-attention#math-object-2

Boundary

Source boundary: shazeer-2019-mqa, ainslie-2023-gqa, dao-2022-flashattention

Check

Run the KV cache lab with one variable moving at a time.

Evidence

Separate cache writes, cache reads, and IO-aware attention.

Next move

First double T and predict the memory change. Then reduce Hkv and predict the cache ratio. Keep FlashAttention out of that prediction until you switch to a prefill/training IO question.

Best explanationSeparate cache writes, cache reads, and IO-aware attention.

KV caching saves recomputing old keys and values, but every decode step still reads the stored K and V tensors. GQA changes the cache-size term Hkv; FlashAttention changes how exact attention is tiled so the full attention matrix is not materialized.

Attach to Efficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2

Student questionIf keys and values are already cached, why does long context still hurt?

The cache removes repeated projection work, not the need to read prior K/V vectors. The first confusion to repair is whether compute was saved, memory traffic was saved, or both.

Learner repair

Professor noteMake the invariant a shape statement before making it a systems claim.

Name B, L, T, Hkv, d_head, and bytes before saying “efficient.” Then ask which symbol moves when context grows, when KV heads are shared, and when the kernel avoids materializing attention.

Teach the chain

Practitioner exampleA serving engineer cares about the term that must be read on every new token.

Hold layers, batch, and head size fixed. Doubling T roughly doubles KV memory. Moving from full MHA to GQA reduces the KV-head factor, but quality, batching, paging, and hardware still decide the final latency.

Production caveat

Source correctionDo not credit FlashAttention for shrinking the decode KV cache.

Use Shazeer and Ainslie for MQA/GQA decode bandwidth and KV-head sharing. Use Dao et al. for IO-aware exact attention and avoiding the full attention matrix. These are related efficiency stories, not the same lever.

Source boundary: shazeer-2019-mqa, ainslie-2023-gqa, dao-2022-flashattention

AI summaryThe tutor prompt should answer from this object, not from generic attention lore.

Ask for one formula, one toy-code witness, one source boundary, one misconception repair, and one next experiment. If the answer merges GQA and FlashAttention into one claim, mark it as ungrounded.

Draft, not canonical

Open questionWhere does the quality tradeoff appear when fewer KV heads serve more query heads?

The room should preserve this as an open research/practice question instead of pretending the cache equation proves model quality. The next evidence needs model, workload, and metric boundaries.

Unresolved by design

Next experimentRun the KV cache lab with one variable moving at a time.

First double T and predict the memory change. Then reduce Hkv and predict the cache ratio. Keep FlashAttention out of that prediction until you switch to a prefill/training IO question.

Prediction before reveal

Grounded AI handoffAsk from the room packet, not generic attention lore.

You are my AI learning companion for Continuous Function. Current context: Efficient Attention at Scale: KV Cache, GQA & FlashAttention object room. Learning surface: Efficient Attention at Scale: KV Cache, GQA & FlashAttention: Efficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2. What this page says: Exact equation object Current section: High-signal object room. Suggested next step: Run the KV cache lab with one variable moving at a time.. Section excerpt: I am working in Continuous Function's research reading room. Object: equation - Efficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2 Object key: equation:attention-transformers/efficient-attention#math-object-2 Context: Mem_{KV} \approx B\cdot N_{layers}\cdot T\cdot H_{kv}\cdot d_{head}\cdot 2 \cdot bytes. Anchor id: equation/concept-notebook/attention-transformers/efficient-attention/math/equation-2 Open question: For Equation 2 in Efficient Attention at Scale: KV Cache, GQA & FlashAttention, what does each symbol mean, what assumption makes it valid, and which source or code witness supports it? Selected object-room context: - Object context: Exact equation object - Source boundary: CF editorial source-scope review - sources: shazeer-2019-mqa, ainslie-2023-gqa Distilled room cards: - Best explanation: Separate cache writes, cache reads, and IO-aware attention. (Attach to Efficient Attention at Scale: KV Cache, GQA & FlashAttention equati...) - KV caching saves recomputing old keys and values, but every decode step still reads the stored K and V tensors. GQA changes the cache-size term Hkv; FlashAttention changes how exact attention is tiled so the full attention matrix is not... - Student question: If keys and values are already cached, why does long context still hurt? (Learner repair) - The cache removes repeated projection work, not the need to read prior K/V vectors. The first confusion to repair is whether compute was saved, memory traffic was saved, or both. - Professor note: Make the invariant a shape statement before making it a systems claim. (Teach the chain) - Name B, L, T, Hkv, d_head, and bytes before saying “efficient.” Then ask which symbol moves when context grows, when KV heads are shared, and when the kernel avoids materializing attention. - Practitioner example: A serving engineer cares about the term that must be read on every new token. (Production caveat) - Hold layers, batch, and head size fixed. Doubling T roughly doubles KV memory. Moving from full MHA to GQA reduces the KV-head factor, but quality, batching, paging, and hardware still decide the final latency. - Source correction: Do not credit FlashAttention for shrinking the decode KV cache. (Source boundary: shazeer-2019-mqa, ainslie-2023-gqa, dao-2022-flashattention) - Use Shazeer and Ainslie for MQA/GQA decode bandwidth and KV-head sharing. Use Dao et al. for IO-aware exact attention and avoiding the full attention matrix. These are related efficiency stories, not the same lever. - AI summary: The tutor prompt should answer from this object, not from generic attention lore. (Draft, not canonical) - Ask for one formula, one toy-code witness, one source boundary, one misconception repair, and one next experiment. If the answer merges GQA and FlashAttention into one claim, mark it as ungrounded. - Open question: Where does the quality tradeoff appear when fewer KV heads serve more query heads? (Unresolved by design) - The room should preserve this as an open research/practice question instead of pretending the cache equation proves model quality. The next evidence needs model, workload, and metric boundaries. - Next experiment: Run the KV cache lab with one variable moving at a time. (Prediction before reveal) - First double T and predict the memory change. Then reduce Hkv and predict the cache ratio. Keep FlashAttention out of that prediction until you switch to a prefill/training IO question. - Next experiment: Run the KV cache lab with one variable moving at a time. First double T and predict the memory change. Then reduce Hkv and predict the cache ratio. Keep FlashAttention out of that prediction until you switch to a prefill/training IO ques... - Local-draft boundary: This is a local distilled object room for tutoring and discussion. Treat it as draft context, not canonical atlas content. Evidence to inspect: - Source ids to inspect: shazeer-2019-mqa, ainslie-2023-gqa, dao-2022-flashattention - Symbol meanings, shapes, units, and hidden assumptions - The source line, page, or concept section where the equation is justified - A runnable code witness that mirrors the same terms Deterministic role lenses for this object: - Boundary: fixed perspectives, not people, community contributions, or independent review - Source-checking summary: Treat this as a symbol-and-shape object: the discussion resolves only when each term has a role, shape, assumption, and witness. Current carried context: Exact equation object - Proposed experiment: Run the KV cache lab with one variable moving at a time. First double T and predict the memory change. Then reduce Hkv and predict the cache ratio. Keep FlashAttention out of that prediction until you switch to a prefill/training IO ques... - Teach/transfer move: Name the invariant in words, then reuse it on the next equation or architecture tradeoff. - Assumptions: - Carried source boundary: CF editorial source-scope review - sources: shazeer-2019-mqa, ainslie-2023-gqa - The stable content-object key lets local drafts, prompts, and route memory attach without changing the source page. - A symbol that appears obvious may be carrying batch, sequence, head, or precision assumptions. - A source equation and a teaching equation may differ; keep that translation visible. - Role-lens requests: - Learner: ask for "Ask what would make "Efficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2" feel predictable rather than familiar." | assumption: Carried source boundary: CF editorial source-scope review - sources: shazeer-2019-mqa, ainslie-2023-gqa | next action: Every symbol has a clear role and shape - Researcher: ask for "Source ids to inspect: shazeer-2019-mqa, ainslie-2023-gqa, dao-2022-flashattention" | assumption: The stable content-object key lets local drafts, prompts, and route memory attach without changing the source page. | next action: The learner knows which term changes in the paper or demo - Experimenter: ask for "Choose one variable or condition to perturb before asking for an explanation." | assumption: A symbol that appears obvious may be carrying batch, sequence, head, or precision assumptions. | next action: The equation predicts the observed behavior without extra hand-waving - Professor: ask for "Find the smallest transferable rule a learner could reuse without the AI." | assumption: A source equation and a teaching equation may differ; keep that translation visible. | next action: Teach or transfer: Name the invariant in words, then reuse it on the next equation or architecture tradeoff. What would resolve this: - Every symbol has a clear role and shape - The learner knows which term changes in the paper or demo - The equation predicts the observed behavior without extra hand-waving Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action. Learner goal: Understand the idea. Preferred explanation style: Visual first. Current stuck reason: Object-room handoff. Task: Use the selected object-room packet to answer as a careful tutor. Start with the best explanation, repair the likely misconception, respect the source boundary, and end with the next experiment. My question: For Equation 2 in Efficient Attention at Scale: KV Cache, GQA & FlashAttention, what does each symbol mean, what assumption makes it valid, and which source or code witness supports it? Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Ask with this object Run prediction Inspect code witness

Route back through the notebook

Carry the same object through intuition, math, code, and demo.

4/4 sections ready

Carry inScaled Dot-Product Attention & Transformer Layers

Bring the mental model from Scaled Dot-Product Attention & Transformer Layers; this page will reuse it instead of restarting from zero.

Work hereEfficient Attention at Scale: KV Cache, GQA & FlashAttention

How attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.

Carry outLLM Serving at Scale: Prefill, Decode & Continuous Batching

The next edge should feel earned: use the demo prediction here before following LLM Serving at Scale: Prefill, Decode & Continuous Batching.

After The First Pass

Turn the concept into an inspected object.

The lower panels are one second act: keep the object fixed, inspect it visually, check source boundaries, practice transfer, then attach the research question.

EquationEfficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2Exact equation object

01MechanismInspect motionSee how the idea changes form before adding more explanation.02VisualChoose a cueBind one visible mark to one mathematical role.03SourcesCheck boundary2 source links attached to the object.04ClaimAudit evidenceCF editorial source-scope review 05PracticeAttempt transferWrite before feedback so the model has something to repair.06RoomAsk groundedAttach the question to the same exact object.

Mechanism Storyboard

See the idea move before the page explains it

How attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.

Demo notes open01 / Intuition

Editorial transformer-systems illustration of KV-cache sharing, grouped query lanes, and reduced memory bandwidth.

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Efficient Attention at Scale: KV Cache, GQA & FlashAttention should make visible.

Visual Inquiry

Make the image answer a mathematical question

How attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.

4/4 stages readyDemo notes connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Efficient Attention at Scale: KV Cache, GQA & FlashAttention easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

Object - EquationEfficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2Question

For Equation 2 in Efficient Attention at Scale: KV Cache, GQA & FlashAttention, what does each symbol mean, what assumption makes it valid, and which source or code witness supp...

equation:attention-transformers/efficient-attention#math-object-2

Boundary

CF editorial source-scope review - sources: shazeer-2019-mqa, ainslie-2023-gqa

Check

Open the closest source note before trusting the local explanation.

Evidence

2 selected-object sources shown first; 3 references total.

Next move

Audit the claim boundary, then ask from the same selected object.

selected object source · paper · 2019Fast Transformer Decoding: One Write-Head is All You NeedShazeerLocated at arXiv:1911.02150v1, §2.4.1 (pp. 3–4) and §3–3.1 (pp. 4–6)

Located CF editorial boundary

Grounds the KV-cache bandwidth motivation for sharing keys and values across query heads during decoding.

Used here as

Shazeer identifies repeated loading of large key/value tensors as an incremental-decoding bottleneck and proposes sharing keys and values across heads; Ainslie et al. present GQA as an in...

Caveat

This checks KV-sharing memory/bandwidth and cache-size scaling for autoregressive decoding only; it is not a universal latency, quality, hardware, checkpoint-conversion, or model-adoption...

Open source

selected object source · paper · 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head CheckpointsAinslie et al.Located at EMNLP 2023, §2.2, pp. 4895–4896 (PDF pp. 1–2)

Located CF editorial boundary

Introduces grouped-query attention as the practical middle point between multi-head quality and MQA memory savings.

Used here as

Shazeer identifies repeated loading of large key/value tensors as an incremental-decoding bottleneck and proposes sharing keys and values across heads; Ainslie et al. present GQA as an in...

Caveat

This checks KV-sharing memory/bandwidth and cache-size scaling for autoregressive decoding only; it is not a universal latency, quality, hardware, checkpoint-conversion, or model-adoption...

Open source

paper · 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessDao et al.

Located CF editorial boundary

Grounds the IO-aware attention kernel: tiling plus online softmax avoids materializing the full attention matrix.

Used here as

Concept-level reference for the mechanism on this page.

Caveat

Attached source metadata is a review boundary, not proof of the local explanation.

Open source

Claim Review

How attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.

Object - EquationEfficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2Question

For Equation 2 in Efficient Attention at Scale: KV Cache, GQA & FlashAttention, what does each symbol mean, what assumption makes it valid, and which source or code witness supp...

equation:attention-transformers/efficient-attention#math-object-2

Boundary

CF editorial source-scope review - sources: shazeer-2019-mqa, ainslie-2023-gqa

Check

Treat every claim as provisional until source support and a local witness agree.

Evidence

1 structured claim check on this concept.

Next move

Run the prediction or practice transfer before asking for a grounded review.

1 CF editorial source-scope review recorded

Publisher-side editorial review is not independent replication. Claims without it still need exact source-support review. 3 references and 3 local witnesses are available for inspection.

Practice notebook

Use the idea, then test it somewhere new

How attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.

AttemptNo learning claim inferred

Object - EquationEfficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2Question

For Equation 2 in Efficient Attention at Scale: KV Cache, GQA & FlashAttention, what does each symbol mean, what assumption makes it valid, and which source or code witness supp...

equation:attention-transformers/efficient-attention#math-object-2

Boundary

CF editorial source-scope review - sources: shazeer-2019-mqa, ainslie-2023-gqa

Check

Use one state from Efficient Attention at Scale: KV Cache, GQA & FlashAttention to explain what changes, why it changes, and which assumption the explanation needs.

Evidence

No learner move yet; no learning state is inferred.

Next move

Write first, use only the help you need, then try a new case without it.

Explain

Use one state from Efficient Attention at Scale: KV Cache, GQA & FlashAttention to explain what changes, why it changes, and which assumption the explanation needs.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Attempt context

Learner move

Write an attempt before asking the companion.

Assistance used

0 of 3 progressive hints opened.

Evidence boundary

This draft and any AI response do not establish mastery; a later unassisted case can.

Misconception check

Grounded object roomClose

Selected object routeAsk from this object; carry one invariant back.CF editorial source-scope review - sources: shazeer-2019-mqa, ainslie-2023-gqa

ObjectEquationEfficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2
PredictBefore revealEfficient Attention at Scale: KV Cache, GQA & FlashAttention prediction
WitnessCompare codeEfficient Attention at Scale: KV Cache, GQA & FlashAttention code wit...
RoomAsk groundedChecking local snapshot

EquationEfficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2Exact equation object

Code witness comparisonEfficient Attention at Scale: KV Cache, GQA & FlashAttention code witness 1def kv_cache_gb( batch=1, layers=32, tokens=4096, kv_heads=32, head_dim=128, bytes_per_value=...Prediction before revealEfficient Attention at Scale: KV Cache, GQA & FlashAttention predictionManipulate one control and predict the visible change.

Grounded room questionFor Equation 2 in Efficient Attention at Scale: KV Cache, GQA & FlashAttention, what does each symbol mean, what assumption makes it valid, and which source or code witness supp...Checking local snapshot

Research Room

Attach the question to an exact object

Pick the concept, equation, source, code witness, claim, misconception, or demo state before asking for help. The handoff stays grounded to that object.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

equationEquation snippet: Efficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2. Source notation: Mem_{KV} \approx B\cdot N_{layers}\cdot T\cdot H_{kv}\cdot d_{head}\cdot 2 \cdot bytes.

Efficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2

Anchored question

For Equation 2 in Efficient Attention at Scale: KV Cache, GQA & FlashAttention, what does each symbol mean, what assumption makes it valid, and which source or code witness supports it?

Source boundaryInspect source ids: shazeer-2019-mqa, ainslie-2023-gqa, dao-2022-flashattentionStable content-object key attached

Role lenses for this object

These are fixed, deterministic perspectives derived from the selected object. They do not represent people, community contributions, or independent review.

Learner evidence requestAsk what would make "Efficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2" feel predictable rather than familiar.

Assumption

Source ids shazeer-2019-mqa, ainslie-2023-gqa, dao-2022-flashattention must support the exact object, not just the surrounding topic.

Source-checking summary

Treat this as a symbol-and-shape object: the discussion resolves only when each term has a role, shape, assumption, and witness.

Proposed experiment

Change the term the learner thinks matters most and compare the predicted effect against the code or demo witness.

Next action

Every symbol has a clear role and shape

Evidence4 checks

PredictionChecking carried observation

ActionReady for one action

AILearner handoff ready

Open source object

01PredictionChecking browser-local route memory

02EvidenceChecking for a carried observation

03BoundaryInspect source ids: shazeer-2019-mqa, ainslie-2023-gqa, dao-2022-flashattention

04Next moveSave one next action

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays locally in this browser for equation:attention-transformers/efficient-attention#math-object-2.

Draft noteNext action

No local draft saved.

Evidence to inspect

Source ids to inspect: shazeer-2019-mqa, ainslie-2023-gqa, dao-2022-flashattention
Symbol meanings, shapes, units, and hidden assumptions
The source line, page, or concept section where the equation is justified
A runnable code witness that mirrors the same terms

What would resolve this

Every symbol has a clear role and shape
The learner knows which term changes in the paper or demo
The equation predicts the observed behavior without extra hand-waving

Object-attached AI handoff

I am working in Continuous Function's research reading room. Object: equation - Efficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2 Object key: equation:attention-transformers/efficient-attention#math-object-2 Context: Mem_{KV} \approx B\cdot N_{layers}\cdot T\cdot H_{kv}\cdot d_{head}\cdot 2 \cdot bytes. Anchor id: equation/concept-notebook/attention-transformers/efficient-attention/math/equation-2 Open question: For Equation 2 in Efficient Attention at Scale: KV Cache, GQA & FlashAttention, what does each symbol mean, what assumption makes it valid, and which source or code witness supports it? Evidence to inspect: - Source ids to inspect: shazeer-2019-mqa, ainslie-2023-gqa, dao-2022-flashattention - Symbol meanings, shapes, units, and hidden assumptions - The source line, page, or concept section where the equation is justified - A runnable code witness that mirrors the same terms Deterministic role lenses for this object: - Boundary: fixed perspectives, not people, community contributions, or independent review - Source-checking summary: Treat this as a symbol-and-shape object: the discussion resolves only when each term has a role, shape, assumption, and witness. - Proposed experiment: Change the term the learner thinks matters most and compare the predicted effect against the code or demo witness. - Teach/transfer move: Name the invariant in words, then reuse it on the next equation or architecture tradeoff. - Assumptions: - Source ids shazeer-2019-mqa, ainslie-2023-gqa, dao-2022-flashattention must support the exact object, not just the surrounding topic. - The stable content-object key lets local drafts, prompts, and route memory attach without changing the source page. - A symbol that appears obvious may be carrying batch, sequence, head, or precision assumptions. - A source equation and a teaching equation may differ; keep that translation visible. - Role-lens requests: - Learner: ask for "Ask what would make "Efficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2" feel predictable rather than familiar." | assumption: Source ids shazeer-2019-mqa, ainslie-2023-gqa, dao-2022-flashattention must support the exact object, not just the surrounding topic. | next action: Every symbol has a clear role and shape - Researcher: ask for "Source ids to inspect: shazeer-2019-mqa, ainslie-2023-gqa, dao-2022-flashattention" | assumption: The stable content-object key lets local drafts, prompts, and route memory attach without changing the source page. | next action: The learner knows which term changes in the paper or demo - Experimenter: ask for "Choose one variable or condition to perturb before asking for an explanation." | assumption: A symbol that appears obvious may be carrying batch, sequence, head, or precision assumptions. | next action: The equation predicts the observed behavior without extra hand-waving - Professor: ask for "Find the smallest transferable rule a learner could reuse without the AI." | assumption: A source equation and a teaching equation may differ; keep that translation visible. | next action: Teach or transfer: Name the invariant in words, then reuse it on the next equation or architecture tradeoff. What would resolve this: - Every symbol has a clear role and shape - The learner knows which term changes in the paper or demo - The equation predicts the observed behavior without extra hand-waving Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action. Current deterministic role lens for this object: - Role lens: Learner - Evidence request: Ask what would make "Efficient Attention at Scale: KV Cache, GQA & FlashAttention equation 2" feel predictable rather than familiar. - Assumption to keep visible: Source ids shazeer-2019-mqa, ainslie-2023-gqa, dao-2022-flashattention must support the exact object, not just the surrounding topic. - Proposed experiment: Change the term the learner thinks matters most and compare the predicted effect against the code or demo witness. - Next action: Every symbol has a clear role and shape

equation/concept-notebook/attention-transformers/efficient-attention/math/equation-2
equation:attention-transformers/efficient-attention#math-object-2

Learning Map

Before / Now / Try / Next

BeforeScaled Dot-Product Attention & Transformer Layers

NowIntuition → Math → Code → Demo

TryManipulate one control and predict the visible change.

NextLLM Serving at Scale: Prefill, Decode & Continuous Batching

Learnerpicture first Researchersource check Experimentertest the demo Professorname invariant

01Predict 02Ground 03Code 04Carry

01Intuitionready
02Mathready
03Codeready
04Interactive Demoready

Object Companion

Ask beside the selected object

How attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Attention & Transformers concept. Learning surface: Efficient Attention at Scale: KV Cache, GQA & FlashAttention. What this page says: How attention becomes practical at long context: KV caching for decoding, grouped-query attention, and IO-aware kernels like FlashAttention. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Copy-only tool: this panel composes a prompt for an AI tool you already use. Nothing you enter here is sent from this page.

Domain

Attention & Transformers

attention-transformersefficient-attentionkv-cacheflashattentiongqa

Prerequisites

Scaled Dot-Product Attention & Transformer Layers Rotary Position Embeddings (RoPE)Efficiency: Quantization, Distillation, LoRA & Sparse MoE

Leads To

LLM Serving at Scale: Prefill, Decode & Continuous Batching Speculative Decoding: Lossless Multi-Token Generation Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization Grouped-Query Attention: Sharing KV Heads

LLM Serving at Scale: Prefill, Decode & Continuous Batching Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization Speculative Decoding: Lossless Multi-Token Generation Grouped-Query Attention: Sharing KV Heads

Within this domain