Continuous FunctionStart learning
FoundationsChecking saved investigationReading browser-local route memory before showing a continuation.

Foundation Lab

Rotary Position Embeddings (RoPE)

GPT-NeoX, Llama 1/2/3, PaLM, Gemini, Claude 3: RoPE is the dominant position encoding for modern LLMs

Concept 18 of 100RepresentationsPhase 2
Open local demoAll Foundations
#18RoPERepresentations
key equation
q~p⊤k~q=q⊤R(θq−θp)k\tilde q_p^\top \tilde k_q = q^\top R(\theta_q - \theta_p) k
Reading map and next steps

Selected Foundation Object

Keep the equation fixed; move through the evidence.

Concept 18 of 100RoPERepresentations / Phase 2: Architecture fundamentals
Current question

Why rotation specifically? Because group composition R(θ_p)^T R(θ_q) = R(θ_q - θ_p) automatically produces relative position

q~p⊤k~q=q⊤R(θq−θp)k\tilde q_p^\top \tilde k_q = q^\top R(\theta_q - \theta_p) k
PredictionCommit before the demo.

Ask what should change when the equation is manipulated, then let the local visualization test that expectation.

EvidenceCompare local witness and source.

Use the local runnable panel, the key equation, and canonical papers as separate witnesses for the same object.

InvariantName what survives notation changes.

The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.

Next moveContinue in the newer notebook.

This atlas page keeps a local demo; the domain notebook carries the fuller Intuition -> Math -> Code -> Demo sequence.

Open local demoInspect equationOpen newer notebookNext foundation

Why It Matters for Modern Models

  • GPT-NeoX, Llama 1/2/3, PaLM, Gemini, Claude 3: RoPE is the dominant position encoding for modern LLMs
  • Enables better length extrapolation than learned absolute positions—models can handle longer contexts than seen in training
  • Multi-frequency structure naturally represents both local patterns (high ω) and long-range dependencies (low ω)

What Tutorials Skip

What is still poorly explained in textbooks and papers:

  • Why rotation specifically? Because group composition R(θ_p)^T R(θ_q) = R(θ_q - θ_p) automatically produces relative position
  • How multi-frequency pairs work like clock hands: fast clocks for nearby tokens, slow clocks for distant ones
  • Why long-context methods (Position Interpolation, YaRN) scale positions—prevents phase wrapping beyond training distribution
  • Geometric picture: RoPE is equivariance to translation, similar to CNNs but for 1D sequences via rotation group

Interactive Visualization

Core Math (Optional Deep Dive)

If you want intuition first, start with the key equation and local visualization. Come back here for the full walkthrough.

Key Equation
q~p⊤k~q=q⊤R(θq−θp)k\tilde q_p^\top \tilde k_q = q^\top R(\theta_q - \theta_p) k

Attention is permutation-equivariant by design—without position encoding, transformers can't distinguish token order.

RoPE encodes position as a rotation applied to queries and keys. For 2D subspace with position pp:

q~p=R(θp)q,k~q=R(θq)k\tilde q_p = R(\theta_p) q, \quad \tilde k_q = R(\theta_q) k

where R(θ)=(cos⁡θ−sin⁡θsin⁡θcos⁡θ)R(\theta) = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}

The key property: rotations compose via relative position:

q~p⊤k~q=q⊤R(θp)⊤R(θq)k=q⊤R(θq−θp)k\tilde q_p^\top \tilde k_q = q^\top R(\theta_p)^\top R(\theta_q) k = q^\top R(\theta_q - \theta_p) k

Full RoPE applies this to multiple 2D pairs at different frequencies ωi=base−2i/d\omega_i = \text{base}^{-2i/d}, creating a multi-scale positional ruler.

In complex notation: q~p=q⋅eiθp\tilde q_p = q \cdot e^{i\theta_p}, making relative position a phase difference.

Canonical Papers

RoFormer: Enhanced Transformer with Rotary Position Embedding

Su et al.2021arXiv
Read paper →

Extending Context Window via Positional Interpolation

Chen et al.2023arXiv
Read paper →

YaRN: Efficient Context Window Extension

Peng et al.2024ICLR
Read paper →

Connections

Prerequisites

⊗Attention

Enables

⚙Efficient Attention📏Long Context

Next Moves

Choose the next question to carry this object forward.

Semantic Connections

Break / Breaks When
📏Phase wraparound→Long Context
Fix / Invented to Fix
📏RoPE scaling→Long Context⊗Position encoding→Attention📏Position extrapolation→Long Context
Reuse / Same Technique
⚙Lazy position compute→Efficient Attention
Map / Analogy
🔀Position via state→SSMs & Hybrids