Rotary Position Embeddings (RoPE)

How do query and key rotations make an attention score depend on relative position?

The idea

RoPE rotates pairs of query and key coordinates by angles proportional to position. Shifting both positions by the same amount turns the query and key by the same extra angle in each pair, which leaves their dot product unchanged, so the score depends on position only through the offset between them.

Builds on Scaled Dot-Product Attention & Transformer Layers.

The picture

Attention compares words by dot products, and a dot product does not know where a word sits. Swap two words and every pair of words still gets the same score as before.

RoPE writes position into the vectors themselves. Before a query and a key meet, each is turned: the word at position $m$ by $m$ times a fixed angle. The figure uses 30° per position, one query $[2, 1]$ and one key $[1, 1]$ .

Turning keeps every length, so the only thing position can change is the angle between the two arrows. Move both words later by the same amount and both arrows turn by the same extra angle: the gap between them, and so the score, stays put. What reaches the score is the relative offset, how far back the key sits.

One speed of turning repeats: at 30° per place, 12 places back looks the same as 0. So a real head turns many pairs of coordinates at many speeds, like clock hands:

fast pairs tell neighbouring words apart,
slow pairs tell distant words apart.

Place two words. Watch what the score can see.

The query is [2, 1] and the key is [1, 1]. Each is turned 30° for every position before they meet in a dot product.

Query positionKey position

1The turned vectors

Each arrow stays on its own circle: turning never changes a length.

2The score, by how far back the key sits

The highlighted bar is the current pair. Move both words together and it does not move.

query turned: 150°
key turned: 60°
key trails by: 90°
lengths: 2.236 · 1.414
score q·k: 1.000

Query at 5, key at 2: the key is 3 places back, so its arrow trails by 90°. Score 1.000. Any two positions 3 apart, in this order, give the same score.

Put both words somewhere new without changing the score.

One pair of coordinates, one speed of turning. A real head turns many pairs at many speeds, each exactly this way.

The mathematics

In a 2D subspace, define a rotation matrix:

R(\theta) = \begin{pmatrix} \cos\theta & -\sin\theta \\\\ \sin\theta & \cos\theta \end{pmatrix}.

Write $m$ for the position of the query and $n$ for the position of the key. RoPE turns the query by the angle $\theta_m$ and the key by $\theta_n$ , and their dot product depends only on the difference of the angles:

\tilde q_m^\top \tilde k_n = q^\top R(\theta_n - \theta_m)\, k, \qquad \tilde q_m = R(\theta_m)\, q, \qquad \tilde k_n = R(\theta_n)\, k, \qquad R(\theta_m)^\top R(\theta_n)=R(\theta_n-\theta_m).

So attention can depend on relative position through $\theta_n-\theta_m$ .

In practice, RoPE applies this to many 2D pairs with different frequencies. The usual choice (Su et al., 2024, §3.2.2) is:

\theta_{m,i} = m\,\omega_i, \qquad \omega_i = \mathrm{base}^{-2i/d},

where $d$ is the head dimension, $i$ indexes the 2D pairs and the base is usually 10,000. With $\theta_{m,i}=m\,\omega_i$ , the angle between a query at $m$ and a key at $n$ in pair $i$ is $(n-m)\,\omega_i$ : only the offset $n-m$ enters.

RoPE applies this rotation independently to each 2D coordinate pair $(2i,2i+1)$ of a head, so $d$ is typically even and $i \in \{0,\dots,\frac d2-1\}$ .

Try it

First a worked example, then one to finish, then problems on your own. Check your answer before you look at the solution. Answers typed here are not kept when you leave.

1 · Worked example
The figure's numbers, by hand
Query $q = [2, 1]$ at position 5; key $k = [1, 1]$ at position 2; 30° per position. Find the score of the turned vectors.
1. Turn the query by $5 \times 30^{\circ} = 150^{\circ}$ : $R(150^{\circ})\,q = [-2.232,\ 0.134]$ . Its length is still $2.236$ .
Say the next step to yourself first.
2 · Finish it
Two places back
Same $q$ and $k$ , but now the query is at position 6 and the key at position 4. The first steps are done. What is the score? Give three decimal places.
1. Gap. The key is 2 places back, so it trails by $2 \times 30^{\circ} = 60^{\circ}$ .
2. Turn the key back by the gap. $R(-60^{\circ})\,k = [\cos 60^{\circ} + \sin 60^{\circ},\ \cos 60^{\circ} - \sin 60^{\circ}] = [1.366,\ -0.366]$ .
3. Your turn.
Your answer
3 · On your own
The score comes back
With the key 0 places back, the score is $q \cdot k = 3$ . How many places back must the key sit, at the fewest (more than 0), for the score to be exactly 3 again?
Your answer
4 · On your own
Nearest is not always best
The same key sits 0, 1, 2 or 3 places before the query. Which placement gives the highest score?
Choose one0 places back, the same position1 place back2 places back3 places back
5 · On your own
A key that comes later
In a model that lets a word attend to words after it, as an encoder does, the same key can sit 3 places after the query: query at position 2, key at position 5. Same $q = [2, 1]$ , $k = [1, 1]$ and 30° per position. What is the score?
Your answer

The code

import numpy as np

def R(theta):
    c, s = np.cos(theta), np.sin(theta)
    return np.array([[c, -s], [s, c]])

def rope_dot(q, k, m, n, w):
    # Query at position m, key at position n, one rotation speed w.
    return float((R(m * w) @ q) @ (R(n * w) @ k))

q = np.array([1.0, 0.2])
k = np.array([0.3, 1.0])
w = 0.7  # one frequency, for illustration

for delta in [0, 1, 2, 4, 8]:
    a = rope_dot(q, k, m=0, n=delta, w=w)
    b = rope_dot(q, k, m=5, n=5 + delta, w=w)  # same offset, both moved 5 places
    print("delta =", delta, "dot =", round(a, 3), "dot (shifted) =", round(b, 3))

Keep it

What does RoPE do to a query or key at position $m$ ?
Answer from memory, then check
It turns each pair of coordinates by the angle $m\theta$ , where $\theta$ is that pair's speed. The value vectors are not turned.
Why does the score of a turned query and key depend only on how far apart they are?
Answer from memory, then check
$(R(m\theta)q)^\top R(n\theta)k = q^\top R((n-m)\theta)\,k$ : turning back by one angle and forward by another is a single turn by the difference.
True or false: RoPE makes vectors at later positions longer.
Answer from memory, then check
False. A rotation keeps every length. Position enters the score only through the angle between the query and the key.
Why does RoPE turn different pairs of coordinates at different speeds?
Answer from memory, then check
One speed repeats: at 30° per place, 12 places back looks the same as 0. Fast pairs separate neighbours; slow pairs separate distant words. In a 64-dimensional head with base 10000, the fastest pair repeats every 6.3 places and the slowest about every 47,000.
Does RoPE make every score fall as two words move apart?
Answer from memory, then check
No. RoFormer shows that a bound on the size of a score, built from all its pairs of coordinates together, tends to fall with distance (Su et al., 2024, §3.4.3). A single score can rise and fall: in this lesson one pair gives 3, 3.098, 2.366 and 1 at gaps 0 to 3, and 3 again at 12.
Every word in a sentence moves 100 places later. What happens to the attention scores between them?
Answer from memory, then check
Nothing, as far as RoPE is concerned: each query and key turns by the same extra angle, so every pair keeps its gap.

Recalling an answer from memory makes it last longer than rereading it (Roediger and Karpicke, 2006). Add these 6 cards and they come back tomorrow, then after gaps of 3, 7, 16, 35 and 75 days while you still remember them. They stay in this browser only.

The edge

The algebra is exact; how far it stretches is not. A model only ever sees the angles its training contexts reach, and past that length its slow pairs turn to angles it has never seen. Models trained with RoPE degrade sharply there (Press et al., 2022; Chen et al., 2023). Position interpolation (Chen et al., 2023) squeezes a longer input's positions back into the trained range, so every angle is a familiar one, then fine-tunes briefly. YaRN (Peng et al., 2024) squeezes unevenly, the slow pairs more than the fast ones that carry local order. Both work up to some multiple of the training length. Whether any fixed rule for position lets a model use contexts far longer than it trained on, or whether something else is needed, is still an open question.

Next thoughtPast its training length, a model meets angles it has never seen, and every earlier word's key must still be stored. What does it take to make attention work at a hundred thousand words?Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization →

The older RoPE lab: two tokens, four rotation speeds, a global shift and a test where you guess the dot product

The picture

The mathematics

Try it

The figure's numbers, by hand

Two places back

The score comes back

Nearest is not always best

A key that comes later

The code

Keep it

The edge