LLM Systems

GPU Memory Accounting

A byte-ledger mental model for what fills GPU memory during LLM training and serving, and why the dominant bucket changes by scenario.

status: reviewimportance: criticaldifficulty 4/5math: graduateread: 18mlive demo

Back to LLM Systems Next: Cost and Latency Observability

Concept Structure

GPU Memory Accounting

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

3prerequisites

3next concepts

3related links

Learner Contract

What this page should let you do.

You are here becauseA byte-ledger mental model for what fills GPU memory during LLM training and serving, and why the dominant bucket changes by scenario.

This LLM Systems concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

Before thisLLM Serving at Scale: Prefill, Decode & Continuous Batching

3 prerequisites listed; refresh them before leaning on the math or code.

LLM Serving at Scale: Prefill, Decode & Continuous Batching Grouped-Query Attention: Sharing KV Heads Practical Training Loop (review)

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextCost and Latency Observability

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Cost and Latency Observability

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.

Claims1/1 reviewed

Sources6 cited

Codeattached

Demolive

Reviewed2026-07-02

Updatedpage 2026-07-02

Learning item flow

4/4 sections readyAsk about this Research room

ConceptGPU Memory AccountingLLM Systems EquationGPU Memory Accounting equation 1Exact equation CodeGPU Memory Accounting runnable code 1Exact runnable code DemoGPU Memory Accounting interactive demoInteractive demo ClaimTraining and serving GPU memory ledgers have different dominant bucke...Exact claim check SourceZeRO: Memory Optimizations Toward Training Trillion Parameter ModelsExact source

ConceptGPU Memory AccountingLLM Systems

6 sources attachedLocal snapshot ready

concept:llm-systems/gpu-memory-accounting

Codenearby Predictbefore reveal Roomitem handoff

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

GPU memory errors are often reported as one blunt fact: "out of memory." That message hides the important question: which object was trying to fit?

Source spine: Rajbhandari et al., ZeRO, Chen et al., Training Deep Nets with Sublinear Memory Cost, Kingma and Ba, Adam, Kwon et al., PagedAttention, PyTorch CUDA memory docs, Hugging Face KV cache docs, and Stanford CS336.

The first move is to stop treating memory as one number. Treat it as a ledger:

Weights: the model parameters you must keep around.
Gradients and optimizer state: training-only state such as gradients, Adam moments, and sometimes a master parameter copy.
Saved activations: intermediate values kept from the forward pass so backward can run.
KV cache: serving-time attention keys and values saved for previous tokens during autoregressive generation.
Temporary buffers: workspaces, communication buffers, kernel scratch space, and framework internals.
Allocator slack: memory reserved by the runtime but not currently occupied by live tensors.

The learner trap is assuming one bucket always dominates. In an Adam training step, optimizer and gradient state can dwarf the weights. In short-prompt inference, weights often dominate. In long-context serving, the KV cache can become the thing that tips the request over the budget.

This page is a conceptual byte ledger. It is not a profiler. The numbers below are deliberately small enough to inspect and deliberately explicit about their assumptions.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1 $M_{\mathrm{weights}} = P b_w.$ Equation 2 $M_{\mathrm{train\ state}} = P(b_w + b_g + b_{\mathrm{opt}}).$

Let $P$ be the number of parameters and $b_w$ be bytes per stored model weight. The parameter memory is

M_{\mathrm{weights}} = P b_w.

During training, a simple Adam-style mixed-precision ledger can include gradients, two moment buffers, and sometimes a full-precision master copy. If $b_g$ is gradient bytes per parameter and $b_{\mathrm{opt}}$ is optimizer-state bytes per parameter, then

M_{\mathrm{train\ state}} = P(b_w + b_g + b_{\mathrm{opt}}).

For a toy Transformer training step with batch size $B$ , sequence length $T$ , hidden width $d$ , layers $L$ , activation dtype bytes $b_a$ , and $s$ saved activation-like tensors per layer, a coarse saved-activation ledger is

M_{\mathrm{act}} \approx B T d L s b_a.

Activation checkpointing changes $s$ : fewer activations are saved, but more forward computation is recomputed during backward. That is why "fits" and "fast" are different questions.

During autoregressive serving, the backward buckets disappear, but the KV cache grows with generated or retained context:

M_{\mathrm{KV}} \approx B L T H_{kv} d_{\mathrm{head}} \cdot 2 b_{\mathrm{kv}}.

The factor of $2$ stores both keys and values. $H_{kv}$ is the number of key/value heads, not necessarily the number of query heads. Grouped-query or multi-query attention changes this ledger by changing $H_{kv}$ .

Finally, the runtime memory you see can be larger than live tensor memory:

M_{\mathrm{reserved}} = M_{\mathrm{allocated}} + M_{\mathrm{allocator\ slack}}.

That distinction matters because an allocator may reserve memory for reuse even when fewer live tensors are currently occupying it.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Runnable code 1def gb(bytes_): return bytes_ / 1e9 def train_ledger(params_b=1.5, layers=24, hidden=2048, ba...python

This witness computes two toy ledgers: one Adam training step and one long-context serving request. It is not a hardware profiler; it just makes every assumption explicit.

def gb(bytes_):
    return bytes_ / 1e9

def train_ledger(params_b=1.5, layers=24, hidden=2048, batch=8, seq=1024):
    p = params_b * 1e9
    weights = p * 2
    gradients = p * 2
    optimizer = p * (4 + 4 + 4)  # master copy + Adam m,v
    activations = batch * seq * hidden * layers * 6 * 2
    temporary = 0.08 * (weights + gradients + optimizer + activations)
    return {
        "weights": gb(weights),
        "optimizer+gradients": gb(optimizer + gradients),
        "activations": gb(activations),
        "temporary": gb(temporary),
    }

def serving_ledger(params_b=7, layers=32, batch=8, tokens=32768, h_kv=8, d_head=128):
    weights = params_b * 1e9 * 2
    kv = batch * layers * tokens * h_kv * d_head * 2 * 2
    temporary = 0.05 * (weights + kv)
    return {"weights": gb(weights), "kv_cache": gb(kv), "temporary": gb(temporary)}

for name, ledger in {"train": train_ledger(), "serve": serving_ledger()}.items():
    winner = max(ledger, key=ledger.get)
    print(name, winner, {k: round(v, 2) for k, v in ledger.items()})

The point is not that these constants are universal. The point is that every memory estimate should say which buckets are included, which dtype was assumed, and whether it is training or serving.

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Before reveal, choose which bucket you think dominates the selected scenario. The lab hides exact GB values, fit verdicts, and allocator slack until you commit.

Live Concept Demo

Explore GPU Memory Accounting

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what GPU Memory Accounting should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

A byte-ledger mental model for what fills GPU memory during LLM training and serving, and why the dominant bucket changes by scenario.

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change GPU Memory Accounting should make visible.

Visual Inquiry

Make the image answer a mathematical question

A byte-ledger mental model for what fills GPU memory during LLM training and serving, and why the dominant bucket changes by scenario.

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make GPU Memory Accounting easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

paper · 2020ZeRO: Memory Optimizations Toward Training Trillion Parameter ModelsRajbhandari et al.

Primary training-memory source for model states, optimizer states, gradients, parameters, activations, temporary buffers, and fragmented memory.

Open source

paper · 2016Training Deep Nets with Sublinear Memory CostChen et al.

Primary activation-checkpointing source for trading saved intermediate activations for recomputation.

Open source

paper · 2015Adam: A Method for Stochastic OptimizationKingma and Ba

Primary optimizer source for first- and second-moment moving averages used by Adam-style memory accounting.

Open source

paper · 2023Efficient Memory Management for Large Language Model Serving with PagedAttentionKwon et al.

Primary serving-memory source for dynamic KV-cache growth, fragmentation, and paged/block allocation.

Open source

documentation · 2026PyTorch CUDA memory APIsPyTorch

Documentation source for the difference between tensor-allocated memory and caching-allocator-reserved memory.

Open source

documentation · 2026Transformers KV cache strategiesHugging Face

Documentation source for dynamic/static/offloaded/quantized KV-cache strategy tradeoffs.

Open source

Claim Review

A byte-ledger mental model for what fills GPU memory during LLM training and serving, and why the dominant bucket changes by scenario.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources6 references

rajbhandari-2020-zero, chen-2016-sublinear-memory, kingma-ba-2015-adam, kwon-2023-pagedattention, pytorch-cuda-memory-212, hf-transformers-kv-cache

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedTraining and serving GPU memory ledgers have different dominant buckets: training tracks weights, gradients, optimizer state, saved activations, temporaries, and allocator slack, while serving tracks weights plus growing KV cache and runtime overhead.Claim metadata: source checked

ZeRO supports the training-state bucket split; Adam supports first/second moment optimizer state; Chen et al. support activation checkpointing as a memory/recompute tradeoff; PagedAttention supports KV-cache growth and fragmentation as serving constraints; PyTorch distinguishes tensor-allocated from allocator-reserved memory; Hugging Face documents cache strategies with memory/latency tradeoffs.

Sources: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, Training Deep Nets with Sublinear Memory Cost, Adam: A Method for Stochastic Optimization, Efficient Memory Management for Large Language Model Serving with PagedAttention, PyTorch CUDA memory APIs, Transformers KV cache strategiesThe page uses toy byte formulas and fixed assumptions. It is not a profiler, hardware sizing guide, vLLM benchmark, PyTorch allocator guarantee, or production OOM predictor.A bounded review summary is present; still check caveats and exact reference scope.

Downloaded and searched ZeRO, activation-checkpointing, PagedAttention, and Adam PDFs plus current PyTorch/Hugging Face docs. The concept is source-supported as a scenario-explicit pedagogical ledger, not as a measured runtime or current serving-stack claim. GPT Pro publication critique remains pending because 127.0.0.1:51672 refused connection.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-02

Source support candidates

paper 2020ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

Primary training-memory source for model states, optimizer states, gradients, parameters, activations, temporary buffers, and fragmented memory.

paper 2016Training Deep Nets with Sublinear Memory Cost

Primary activation-checkpointing source for trading saved intermediate activations for recomputation.

paper 2015Adam: A Method for Stochastic Optimization

Primary optimizer source for first- and second-moment moving averages used by Adam-style memory accounting.

paper 2023Efficient Memory Management for Large Language Model Serving with PagedAttention

Primary serving-memory source for dynamic KV-cache growth, fragmentation, and paged/block allocation.

Mechanism checks

Equation 1

M_{\mathrm{weights}} = P b_w.

Equation 2

M_{\mathrm{train\ state}} = P(b_w + b_g + b_{\mathrm{opt}}).

Runnable code 1def gb(bytes_): return bytes_ / 1e9 def train_ledger(params_b=1.5, layers=24, hidden=2048, ba...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

A byte-ledger mental model for what fills GPU memory during LLM training and serving, and why the dominant bucket changes by scenario.

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in GPU Memory Accounting.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Grounded research drawerClose

ConceptGPU Memory AccountingLLM Systems

Runnable code comparisonGPU Memory Accounting runnable code 1def gb(bytes_): return bytes_ / 1e9 def train_ledger(params_b=1.5, layers=24, hidden=2048, ba...Prediction before revealGPU Memory Accounting interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes GPU Memory Accounting click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptLLM Systems

GPU Memory Accounting

Attached question

What is the smallest example that makes GPU Memory Accounting click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays in this browser, attached to the selected learning item.

Draft noteNext action

No local draft saved.

Evidence to inspect

References to inspect: attached references on this page.
Definition, prerequisite, and contrast concept links
The equation or runnable code that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - GPU Memory Accounting Selected item key: recorded for copy. Context: LLM Systems Page anchor: recorded for copy. Open question: What is the smallest example that makes GPU Memory Accounting click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context

concept/concept-notebook/llm-systems/gpu-memory-accounting
concept:llm-systems/gpu-memory-accounting

Learning Map

Before / Now / Try / Next

BeforeLLM Serving at Scale: Prefill, Decode & Continuous Batching

NowIntuition -> Math -> Code -> Demo

TryManipulate one control and predict the visible change.

NextCost and Latency Observability

Intuitionready
Mathready
Codeready
Interactive Demoready

AI Companion

Ask beside the selected object

A byte-ledger mental model for what fills GPU memory during LLM training and serving, and why the dominant bucket changes by scenario.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: LLM Systems concept. Learning surface: GPU Memory Accounting. What this page says: A byte-ledger mental model for what fills GPU memory during LLM training and serving, and why the dominant bucket changes by scenario. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

LLM Systems

llm-systemsservingtraininggpu-memorykv-cachesystems

Prerequisites

LLM Serving at Scale: Prefill, Decode & Continuous Batching Grouped-Query Attention: Sharing KV Heads Practical Training Loop (review)

Leads To

Cost and Latency Observability Inference Kernels, KV Cache Layout, and Paged Attention (review)Transformer Implementation From Scratch (planned)

Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization FlashAttention: IO-Aware Attention Experiment Tracking (review)

Within this domain