Continuous Function

Foundation reference

Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization

Rescaling RoPE frequencies lets a pretrained model use a longer context after a short fine-tune instead of full retraining; position interpolation, YaRN and LongRoPE are published methods of this kind (Chen et al., 2023; Peng et al., 2024; Ding et al., 2024).

A newer notebook covers this concept. This older page keeps its summary, math and sources.

Concept 30 of 100EfficiencyStudy phase 6
Open the demoOpen the newer notebookAll concepts
Key equation
MemoryKV=2×Nlayers×Hkv×T×dh×bytes\text{Memory}_{KV} = 2 \times N_{\text{layers}} \times H_{kv} \times T \times d_h \times \text{bytes}
Reading map and next steps

Study guide

Where to start

Concept 30 of 100Long ContextEfficiency · Study phase 6: Modern efficiency & inference
Question

Why does a model fail beyond its training length because of unseen rotation angles rather than because it "forgets"?

Open the demoGo to the mathOpen the newer notebookNext concept
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.

Before you move a control in the demo, say what you expect to change.

CheckCheck against the math and the papers.

Compare what the demo shows with the key equation and with the papers cited below.

GeneralizeName what stays the same.

Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.

ContinueContinue in the newer notebook.

The newer notebook covers the same concept and has an interactive demo.

Why It Matters for Modern Models

  • Rescaling RoPE frequencies lets a pretrained model use a longer context after a short fine-tune instead of full retraining; position interpolation, YaRN and LongRoPE are published methods of this kind (Chen et al., 2023; Peng et al., 2024; Ding et al., 2024).
  • At long contexts the KV cache becomes the memory bottleneck: KVQuant quantizes it to about 3 bits and reports serving LLaMA-7B with a context of up to 1 million tokens on a single A100-80GB GPU (Hooper et al., 2024).
  • Failures beyond the training length are largely a matter of unseen rotation angles: low-frequency RoPE dimensions reach angles they never reached during training.
  • Long context builds directly on RoPE and on KV caching: the first decides whether distant positions can be told apart, the second whether their keys and values fit in memory.
  • When generating at long contexts, reading the KV cache from memory, rather than arithmetic, often sets cost and throughput, which is why cache quantization and compression matter in practice.

Questions to Think About

  • Apart from position extrapolation, how does a growing number of keys change the softmax, spreading attention more thinly as the context grows?
  • Why should high-frequency RoPE dimensions, which encode fine local order, be rescaled less than low-frequency ones, which encode coarse position?
  • What does quantizing or evicting KV-cache entries cost in long-range retrieval accuracy, and when is that trade worth making?
  • Why is processing a long prompt (prefill) mostly limited by computation while generating each new token is mostly limited by memory bandwidth, and how does a long context change the second?

Interactive Demos

Core Math

If you prefer intuition first, try the demos above, then come back for the math.

Key Equation
MemoryKV=2×Nlayers×Hkv×T×dh×bytes\text{Memory}_{KV} = 2 \times N_{\text{layers}} \times H_{kv} \times T \times d_h \times \text{bytes}

Extending a model's context window raises two separate problems: positions beyond those seen in pretraining, and a key–value (KV) cache whose memory grows with every token.

RoPE as a rotation per frequency:

Pair the coordinates of each query and key vector into two-dimensional blocks and treat block jj as a complex number. RoPE (rotary position embedding) rotates block jj at position mm by the angle mθjm\theta_j:

q~m,j=qm,j eimθj,k~n,j=kn,j einθj,θj=b−2j/d\tilde q_{m,j} = q_{m,j}\, e^{i m\theta_j}, \quad \tilde k_{n,j} = k_{n,j}\, e^{i n\theta_j}, \quad \theta_j = b^{-2j/d}

The attention score depends on the positions only through (m−n)θj(m-n)\theta_j, the relative offset. Block jj has wavelength λj=2π/θj\lambda_j = 2\pi/\theta_j. Low-frequency blocks have wavelengths longer than the training context, so beyond it they produce angles the model never saw.

Position interpolation and YaRN:

Position interpolation divides every position by the extension factor ss, which keeps all angles inside the trained range but squeezes nearby positions together (Chen et al., 2023). YaRN's "NTK-by-parts" rule interpolates only the low-frequency blocks:

λ^j=λj(1−γj)/s+γj,γj={1λj<L/β0λj>L/αL/λj−αβ−αotherwise\hat{\lambda}_j = \frac{\lambda_j}{(1-\gamma_j)/s + \gamma_j}, \quad \gamma_j = \begin{cases} 1 & \lambda_j < L/\beta \\ 0 & \lambda_j > L/\alpha \\ \frac{L/\lambda_j - \alpha}{\beta - \alpha} & \text{otherwise} \end{cases}

Here LL is the original context length and α<β\alpha < \beta set the ends of the ramp. High-frequency blocks (γj=1\gamma_j = 1) keep their wavelengths and so their fine local resolution; low-frequency blocks (γj=0\gamma_j = 0) are stretched by ss. YaRN also rescales the attention logits by a temperature that depends on ss (Peng et al., 2024).

KV cache memory for one sequence:

MemoryKV=2×Nlayers×Hkv×T×dh×bytes per element\text{Memory}_{KV} = 2 \times N_{\text{layers}} \times H_{kv} \times T \times d_h \times \text{bytes per element}

with NlayersN_{\text{layers}} layers, HkvH_{kv} key–value heads, TT tokens and head dimension dhd_h; the factor 2 counts keys and values. Memory grows linearly in TT, so at long contexts the cache, not the weights, limits how many sequences fit on a device.

Key Papers

Extending Context Window of Large Language Models via Positional Interpolation

Chen et al.2023arXiv
Read paper →

YaRN: Efficient Context Window Extension of Large Language Models

Peng et al.2024ICLR
Read paper →

LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

Ding et al.2024ICML
Read paper →

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

Hooper et al.2024NeurIPS
Read paper →

Connections

Prerequisites

RoPEEfficient AttentionLLM ServingAttention

Leads to

State Space ModelsRAGFlashAttentionGQAInfinite Context

Where to go next

A newer notebook covers Long Context, with an interactive demo.

Continue in the newer notebook
Other connections (11)

Which limitations does it address, or leave to a later idea?

  • Fixed-state memory: a limitation of Long Context that State Space Models addresses. Open State Space Models

    Even with RoPE scaling and KV compression, transformer decoding still pays KV costs that grow with T. SSM/hybrid architectures were invented to get fixed-size state and linear-time sequence modeling for very long contexts.

  • Position extrapolation: a limitation of Long Context that RoPE addresses. Open RoPE

    Learned absolute position embeddings have no entry for positions beyond the training length, and RoPE models also degrade past it (Press et al., 2022). Rescaling RoPE's angles, by position interpolation (Chen et al., 2023) or YaRN (Peng et al., 2024), followed by brief fine-tuning, lets models trained on a few thousand tokens use much longer contexts.

  • RoPE scaling: a limitation of RoPE that Long Context addresses. Open RoPE

    RoPE encodes relative position, but models that use it still break down beyond the lengths seen in training. RoPE scaling methods such as YaRN and LongRoPE rescale the rotation frequencies to extend the usable context (Peng et al., 2024; Ding et al., 2024).

  • KV compression: a limitation of Efficient Attention that Long Context addresses. Open Efficient Attention

    FlashAttention removes the quadratic memory cost of attention (the computation is still quadratic), but during generation the key-value cache still grows linearly with context length T. Long-context systems add cache quantization and compression to reduce that remaining cost.

Where does it break down?

  • Attention dilution: where Attention breaks down, taken up on this page. Open Attention

    Even if memory is solved, long contexts make retrieval harder: softmax has to discriminate among many keys, and attention can become diffuse or distracted. Quality can degrade with length, not just speed.

  • Unseen rotation angles: where RoPE breaks down, taken up on this page. Open RoPE

    RoPE rotates each pair of dimensions at its own frequency. The slowest-rotating pairs do not complete a full turn within the training length, so positions beyond it produce rotation angles the model has never seen; this is a main reason RoPE models fail beyond their training length (Peng et al., 2024).

  • Token count grows: where Tokenization breaks down, taken up on this page. Open Tokenization

    Context length is measured in tokens, not characters. Text that splits into many small pieces (rare Unicode, code identifiers, byte-level fallbacks) inflates the token count and uses up the context window and the key-value cache faster than expected.

  • KV cache dominates: where LLM Serving breaks down, taken up on this page. Open LLM Serving

    As prompts get longer, reading and storing the key-value cache dominates the cost of serving. That shrinks the batch sizes that fit in memory and lowers the number of requests served within latency targets; a system tuned for 4,000-token prompts can fail at 128,000.

  • Long patch sequences: where Multimodal Pretraining breaks down, taken up on this page. Open Multimodal Pretraining

    Images and videos become long sequences of patch or frame tokens, often far longer than the text that accompanies them, so they use up the context window and the key-value cache much faster than text alone.

Where else does the same idea appear?

  • Memory efficiency: the same technique appears in Efficient Attention. Open Efficient Attention

    Long context requires efficient attention: GQA reduces KV cache, sliding window limits active tokens, both enable longer sequences within memory constraints.

  • Memory paging strategies: the same technique appears in MoE Serving. Open MoE Serving

    Both MoE expert caching and long-context KV caching use similar memory management: page in/out based on access patterns, predict future needs, trade latency for capacity. PagedAttention borrows from OS memory management.