The bytes of the key and value tensors, from the equation shown, in decimal GB. Every number on this page is exact arithmetic you can redo by hand.
KV-cache memory
The KV-cache challenge
Two questions you can check by hand, then a calculator for every factor. Every number here is exact arithmetic.
One question, before anything is revealed
Hold batch, layers, sequence length, head dimension, and precision fixed. Query heads stay at 32. Cached key/value heads fall from 32 to 8. What is the exact KV-cache ratio?
- Batch
- 1
- Layers
- 32
- Sequence length
- 8,192
- Head dimension
- 128
- Precision
- fp16/bf16
The same model as the site's worked KV-cache reference, at twice its sequence length (8,192 positions instead of 4,096), so the numbers here are new.
What this page is and is not
Nothing on this page is a benchmark: no latency, throughput, memory overhead or model quality was measured.
In this browser only. They are not sent anywhere, and no account or email is needed. Clearing this site's data in your browser removes them.
Open the KV memory step in the LabA prediction, the tensor shapes, the same calculation in Python, and a comparison you can save.Read the KV-cache referenceThe equation, worked numbers, common misconceptions and sources, with no questions first. It works the same model at 4,096 positions.Browse the index of conceptsWhere this equation sits among the rest of the material.