Ask what should change when the equation is manipulated, then let the visualization test that expectation.
Foundation Lab
Rotary Position Embeddings (RoPE)
GPT-NeoX, Llama 1/2/3, PaLM, Gemini, Claude 3: RoPE is the dominant position encoding for modern LLMs
#18RoPERepresentations
key equation
\tilde q_p^\top \tilde k_q = q^\top R(\theta_q - \theta_p) kSelected Foundation Object
Keep the equation fixed; move through the evidence.
Concept 18 of 100RoPERepresentations / Phase 2: Architecture fundamentals
Current question
Why rotation specifically? Because group composition R(θ_p)^T R(θ_q) = R(θ_q - θ_p) automatically produces relative position
\tilde q_p^\top \tilde k_q = q^\top R(\theta_q - \theta_p) kUse the runnable panel, the key equation, and canonical papers as separate forms of evidence for the same object.
The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.
This atlas page keeps the working demo; the domain notebook carries the fuller Intuition -> Math -> Code -> Demo sequence.
Why It Matters for Modern Models
- GPT-NeoX, Llama 1/2/3, PaLM, Gemini, Claude 3: RoPE is the dominant position encoding for modern LLMs
- Enables better length extrapolation than learned absolute positions—models can handle longer contexts than seen in training
- Multi-frequency structure naturally represents both local patterns (high ω) and long-range dependencies (low ω)
What Tutorials Skip
What is still poorly explained in textbooks and papers:
- Why rotation specifically? Because group composition R(θ_p)^T R(θ_q) = R(θ_q - θ_p) automatically produces relative position
- How multi-frequency pairs work like clock hands: fast clocks for nearby tokens, slow clocks for distant ones
- Why long-context methods (Position Interpolation, YaRN) scale positions—prevents phase wrapping beyond training distribution
- Geometric picture: RoPE is equivariance to translation, similar to CNNs but for 1D sequences via rotation group
Interactive Visualization
Core Math (Optional Deep Dive)
If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.
Key Equation
Attention is permutation-equivariant by design—without position encoding, transformers can't distinguish token order.
RoPE encodes position as a rotation applied to queries and keys. For 2D subspace with position :
where
The key property: rotations compose via relative position:
Full RoPE applies this to multiple 2D pairs at different frequencies , creating a multi-scale positional ruler.
In complex notation: , making relative position a phase difference.