Rotary position embeddings (RoPE) encode token position by rotating query and key vectors. RoPE is the position encoding of many large language models, including GPT-NeoX-20B (Black et al., 2022), PaLM (Chowdhery et al., 2023) and the Llama family (Touvron et al., 2023).
A newer notebook covers this concept. This older page keeps its summary, math and sources.
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueContinue in the newer notebook.
The newer notebook covers the same concept and has an interactive demo.
Why It Matters for Modern Models
Rotary position embeddings (RoPE) encode token position by rotating query and key vectors. RoPE is the position encoding of many large language models, including GPT-NeoX-20B (Black et al., 2022), PaLM (Chowdhery et al., 2023) and the Llama family (Touvron et al., 2023).
RoPE models do not extrapolate well past their training length; perplexity rises sharply beyond it (Press et al., 2022; Chen et al., 2023). Position Interpolation and YaRN extend the context window by rescaling positions or frequencies, followed by a short fine-tuning run.
Fast-rotating pairs distinguish nearby offsets, while slow-rotating pairs change little over short distances and vary only across long ones.
Questions to Think About
How do the different frequencies act like the hands of a clock, with fast pairs resolving nearby tokens and slow pairs distinguishing distant ones?
Why do Position Interpolation and YaRN rescale positions or frequencies, and what goes wrong when a model meets rotation angles it never saw during training?
Shifting the query and key positions by the same amount leaves the RoPE score unchanged; how does this compare with the translation equivariance of convolutions?
Interactive Demo
Core Math
If you prefer intuition first, try the demo above, then come back for the math.
Key Equation
q~m⊤k~n=q⊤R((n−m)ω)k
Self-attention without position information is permutation-equivariant: reorder the input tokens and the outputs are reordered the same way, so the layer itself cannot tell token order. (A causal mask gives decoder-only models some implicit position signal; Haviv et al., 2022.)
RoPE encodes position as a rotation applied to queries and keys. In one 2D subspace, a query at position m and a key at position n become
q~m=R(mω)q,k~n=R(nω)k
where R(θ)=(cosθsinθ−sinθcosθ) and ω is the pair's frequency.
Because R(θ)⊤=R(−θ) and rotations compose by adding angles, the score depends only on the offsetn−m:
q~m⊤k~n=q⊤R(mω)⊤R(nω)k=q⊤R((n−m)ω)k
Full RoPE splits each head into d/2 pairs with frequencies ωi=base−2i/d (i=0,…,d/2−1; base 10,000 in the original paper), so different pairs measure offsets on different scales.
In complex notation, each pair is a complex number and q~m=qeimω, so relative position becomes a phase difference.
Key Papers
RoFormer: Enhanced Transformer with Rotary Position Embedding
Which limitations does it address, or leave to a later idea?
RoPE scaling: a limitation of RoPE that Long Context addresses. Open Long Context
RoPE encodes relative position, but models that use it still break down beyond the lengths seen in training. RoPE scaling methods such as YaRN and LongRoPE rescale the rotation frequencies to extend the usable context (Peng et al., 2024; Ding et al., 2024).
Position encoding: a limitation of Attention that RoPE addresses. Open Attention
The original Transformer added sinusoidal or learned absolute position vectors to its inputs. RoPE instead rotates queries and keys so that their dot product depends only on the offset between positions (Su et al., 2024). It does not by itself extrapolate well beyond the training length; that needed later rescaling methods such as position interpolation (Chen et al., 2023).
Position extrapolation: a limitation of Long Context that RoPE addresses. Open Long Context
Learned absolute position embeddings have no entry for positions beyond the training length, and RoPE models also degrade past it (Press et al., 2022). Rescaling RoPE's angles, by position interpolation (Chen et al., 2023) or YaRN (Peng et al., 2024), followed by brief fine-tuning, lets models trained on a few thousand tokens use much longer contexts.
Where does it break down?
Unseen rotation angles: where RoPE breaks down, taken up in Long Context. Open Long Context
RoPE rotates each pair of dimensions at its own frequency. The slowest-rotating pairs do not complete a full turn within the training length, so positions beyond it produce rotation angles the model has never seen; this is a main reason RoPE models fail beyond their training length (Peng et al., 2024).
Where else does the same idea appear?
Positions at attention time: the same technique appears in Efficient Attention. Open Efficient Attention
RoPE rotates each query and key by an angle set by its position just before the attention scores are computed, so no position embeddings are added to the inputs. Because only queries and keys change, it combines directly with FlashAttention and sliding-window attention.
SSMs encode position implicitly through state evolution, RoPE encodes position explicitly through rotation. Both solve "where am I in the sequence?" differently.