∇Continuous Function
FoundationsChecking saved investigationReading browser-local route memory before showing a continuation.

Foundation Lab

Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization

RoPE scaling methods (YaRN, LongRoPE) are dominant route to extending context windows without architecture change—how 32k→128k→256k→1M+ happens

Concept 30 of 100EfficiencyPhase 6
Open New NotebookAll Foundations
#30Long ContextEfficiency
key equation\text{Memory}_{KV} = 2 \times L \times H \times T \times d_h \times \text{bytes}

Selected Foundation Object

Keep the equation fixed; move through the evidence.

Concept 30 of 100Long ContextEfficiency / Phase 6: Modern efficiency & inference
Current question

Failure mode isn't "model forgets"—it's phase/angle OOD: RoPE uses wavelengths, beyond training range you hit unseen rotations

\text{Memory}_{KV} = 2 \times L \times H \times T \times d_h \times \text{bytes}
PredictionCommit before the demo.

Ask what should change when the equation is manipulated, then let the visualization test that expectation.

EvidenceCompare local witness and source.

Use the runnable panel, the key equation, and canonical papers as separate forms of evidence for the same object.

InvariantName what survives notation changes.

The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.

Next moveContinue in the newer notebook.

This atlas page keeps the working demo; the domain notebook carries the fuller Intuition -> Math -> Code -> Demo sequence.

Open demoInspect equationOpen newer notebookNext foundation

Why It Matters for Modern Models

  • RoPE scaling methods (YaRN, LongRoPE) are dominant route to extending context windows without architecture change—how 32k→128k→256k→1M+ happens
  • KV cache becomes bottleneck at very long contexts—KVQuant enables ~1M context on single A100-80GB via quantization to 3-4 bits
  • Position encoding extrapolation is "angle OOD"—RoPE dimensions have wavelengths, beyond training you hit unseen rotations ("critical dimensions")
  • After RoPE basics (#18) and KV cache/FlashAttention (#19), this explains how frontier models actually break the pretraining ceiling
  • Memory math becomes destiny—at long contexts, KV storage and bandwidth dominate compute; quantization/compression is not optional

What Tutorials Skip

What is still poorly explained in textbooks and papers:

  • Failure mode isn't "model forgets"—it's phase/angle OOD: RoPE uses wavelengths, beyond training range you hit unseen rotations
  • Long context breaks attention in two ways: (a) position encoding extrapolation, (b) attention entropy/softmax temperature drift as T grows
  • Non-uniform scaling is key insight—high-frequency RoPE dims (fine detail) need less rescaling than low-frequency (global position)
  • KV compression tradeoff: quantization/pruning saves memory but degrades long-range retrieval—you're trading capacity for length
  • Serving implications: KV cache memory >> compute at long contexts, changes deployment economics (memory-bound not compute-bound)

Interactive Visualization

Core Math (Optional Deep Dive)

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation
MemoryKV=2×L×H×T×dh×bytes\text{Memory}_{KV} = 2 \times L \times H \times T \times d_h \times \text{bytes}

Long context is a two-front war: (1) position extrapolation beyond pretraining, and (2) KV cache memory explosion. Frontier models use RoPE scaling + aggressive KV compression.

RoPE injection (complex form, key idea: relative phase):

qm,[2j:2j+1]=Wqxm⋅eimθj,kn,[2j:2j+1]=Wkxn⋅einθj,θj=b−2j/dq_{m,[2j:2j+1]} = W_q x_m \cdot e^{i m\theta_j}, \quad k_{n,[2j:2j+1]} = W_k x_n \cdot e^{i n\theta_j}, \quad \theta_j = b^{-2j/d}

Rotation encodes position; attention sees (m−n)θj(m-n)\theta_j (relative).

YaRN "NTK-by-parts" wavelength scaling:

λ^j=(1−γj)sλj+γjλj,γj={1λj<L/β0λj>L/αL/λj−αβ−αotherwise\hat{\lambda}_j = (1-\gamma_j)s\lambda_j + \gamma_j\lambda_j, \quad \gamma_j = \begin{cases} 1 & \lambda_j < L/\beta \\ 0 & \lambda_j > L/\alpha \\ \frac{L/\lambda_j - \alpha}{\beta - \alpha} & \text{otherwise} \end{cases}

Non-uniform scaling: keep high-frequency dims, interpolate low-frequency → better extrapolation.

KV cache memory vs sequence length:

Memory=2×L×H×T×dh×bytes/element\text{Memory} = 2 \times L \times H \times T \times d_h \times \text{bytes/element}

where LL layers, HH heads, TT tokens, dhd_h head dimension—grows linearly in TT, dominates at long contexts.

Canonical Papers

YaRN: Efficient Context Window Extension of Large Language Models

Peng et al.2024ICLR
Read paper →

LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

Ding et al.2024ICML
Read paper →

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

Hooper et al.2024arXiv
Read paper →

Connections

Prerequisites

↻RoPE⚙Efficient Attention⚡LLM Serving⊗Attention

Enables

🔀SSMs & Hybrids🔤Tokens🔍RAG⚡FlashAttn👥GQA♾️InfCtx

Next Moves

Choose the next question to carry this object forward.

Semantic Connections

Fix / Invented to Fix
🔀Fixed-state memory→SSMs & Hybrids↻Position extrapolation→RoPE↻RoPE scaling→RoPE⚙KV compression→Efficient Attention
Reuse / Same Technique
⚙Memory efficiency→Efficient Attention⚡Memory paging strategies→MoE Serving
Break / Breaks When
⊗Attention dilution→Attention↻Phase wraparound→RoPE🔤Token explosion→Tokens⚡KV dominates SLO→LLM Serving🖼️Patch sequences explode→Multimodal VLP