Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization

Sliding window attention (reduce $T^2$ compute)

If each token attends only to itself and the $W-1$ tokens before it, a window of $W$ keys (Beltagy et al., 2020), then attention compute is closer to $O(TW)$ instead of $O(T^2)$. One way to express this is with a mask:

and:

KV cache memory (the bottleneck at long $T$)

Across a batch of $B$ sequences and $L$ layers, storing keys and values for $T$ tokens costs roughly:

Here $H_{kv}$ is the number of key/value heads per layer and $d_{\mathrm{head}}$ their width. This grows linearly in $T$. At long context and large batch it can exceed the memory taken by the model's weights: with the settings in the code below (80 layers, 8 KV heads of width 128, batch 8, 2 bytes per value), it reaches about 344 GB at 131,072 tokens.

Three claims about one request

"Supports 131,072 tokens" can mean three things, and each needs its own evidence.

1. The cache fits. For one request ($B=1$) with $N_{\mathrm{layers}}=32$, $T=131{,}072$, $H_{kv}=8$, $d_{\mathrm{head}}=128$ and 2 bytes per value, the formula gives $1\cdot32\cdot131{,}072\cdot8\cdot128\cdot2\cdot2=2^{34}$ bytes, exactly 16 GiB or about 17.18 GB. Halving $H_{kv}$ to 4 halves it. That compares two architectures; whether the narrower one keeps its quality is measured separately (Ainslie et al., 2023). A server allocates more than this ideal count (paging metadata, padding, other tensors), and a measured fit holds for that request, model and implementation.
2. Positions past the training length still work. This needs the same model evaluated beyond its training length, with and without a position method such as interpolation, on the same tasks. A smaller cache says nothing about it.
3. The model finds and uses what it needs. This needs behaviour tests. RULER (Hsieh et al., 2024) adds several planted facts, tracing and aggregation to the single-fact test, and finds that models scoring almost perfectly on that test still lose much of their accuracy as inputs grow. Liu et al. (2024) find that accuracy is often highest when the relevant passage is near the start or end of the input and drops when it is in the middle.

These are not the only constraints; prefill compute, decode bandwidth and scheduling sit beside them. The practice section works the first claim exactly and matches the other two to their evidence.

RoPE scaling (positions beyond the training range)

RoPE encodes position by rotating queries and keys. At positions far beyond training, those rotations reach angles the model never saw, especially in the slowly rotating (low-frequency) pairs, which never completed a full turn during training. RoPE scaling methods change the map from position $m$ to angle $\theta(m)$ so that long inputs produce familiar angles while short-range detail is preserved. Position interpolation (Chen et al., 2023) replaces $m$ by $m\,T_{\mathrm{train}}/T_{\mathrm{new}}$ for every pair, where $T_{\mathrm{train}}$ is the training context length and $T_{\mathrm{new}}$ the extended one; YaRN (Peng et al., 2024) rescales the slow pairs more than the fast ones.