Attention & Transformers

Rotary Position Embeddings (RoPE)

A positional encoding that rotates queries and keys so attention depends on relative position via phase differences.

status: publishedimportance: importantdifficulty 3/5math: undergraduateread: 14mlive demo

Back to Attention & Transformers Next: Efficient Attention at Scale: KV Cache, GQA & FlashAttention

Editorial transformer illustration of rotary position vectors, relative phase arcs, and query-key geometry.

Concept Structure

Rotary Position Embeddings (RoPE)

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

1prerequisites

2next concepts

3related links

Learning map

Rotary Position Embeddings (RoPE)

BeforeScaled Dot-Product Attention & Transformer LayersNow4/4 sections readyTryManipulate one control and predict the visible change.NextEfficient Attention at Scale: KV Cache, GQA & FlashAttention

Object flow

4/4 sections readyAsk about this Research room

ConceptRotary Position Embeddings (RoPE)Attention & TransformersEquationRotary Position Embeddings (RoPE) equation 1Exact equation object CodeRotary Position Embeddings (RoPE) code witness 1Exact code witness DemoRotary Position Embeddings (RoPE) interactive demoVisualization object ClaimRoPE encodes absolute token positions by rotating query and key vecto...Exact claim check SourceRoFormer: Enhanced Transformer with Rotary Position EmbeddingExact source object

ConceptRotary Position Embeddings (RoPE)Attention & Transformers

1 source attachedLocal snapshot ready

concept:attention-transformers/rope

Codewitness nearby Predictbefore reveal Roomobject handoff

Conceptual Bridge

What should feel connected as you move through this page.

Carry inScaled Dot-Product Attention & Transformer Layers

Bring the mental model from Scaled Dot-Product Attention & Transformer Layers; this page will reuse it instead of restarting from zero.

Work hereRotary Position Embeddings (RoPE)

A positional encoding that rotates queries and keys so attention depends on relative position via phase differences.

Carry outEfficient Attention at Scale: KV Cache, GQA & FlashAttention

The next edge should feel earned: use the demo prediction here before following Efficient Attention at Scale: KV Cache, GQA & FlashAttention.

Test the linkManipulate one control and predict the visible change.Then continue to Efficient Attention at Scale: KV Cache, GQA & FlashAttention

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

Self-attention by itself does not know token order: it only sees a set of vectors and compares them.

RoPE injects position by rotating each token's query and key vectors by an angle that depends on its absolute position. That sounds absolute at first: token $p$ gets one rotation, token $q$ gets another.

The key step is what happens when attention compares them. A dot product between two rotated vectors does not keep both rotations separately. The shared frame cancels, leaving the gap between their angles:

rotate the query by position $p$ ,
rotate the key by position $q$ ,
compare them, and the score can see the phase difference $\theta_q-\theta_p$ .

So RoPE uses absolute rotations as the coordinate system, but the query-key score receives a relative-position signal.

A good mental model is "clock hands at multiple speeds":

high-frequency rotations capture local order (nearby tokens),
low-frequency rotations capture long-range order (far-apart tokens).

The careful caveat is that real attention scores still depend on the token content inside $q_p$ and $k_q$ . RoPE does not make every equal-distance token pair have the same attention score; it makes the positional part of the comparison available as a relative phase.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1R(\theta) = \begin{pmatrix} \cos\theta -\sin\theta \\; \sin\theta \cos\theta \end{pmatrix}.Equation 2\tilde q_p = R(\theta_p)q_p,\qquad \tilde k_q = R(\theta_q)k_q.

In a 2D subspace, define a rotation matrix:

R(\theta) = \begin{pmatrix} \cos\theta & -\sin\theta \\\\ \sin\theta & \cos\theta \end{pmatrix}.

RoPE rotates queries and keys by position-dependent angles:

\tilde q_p = R(\theta_p)q_p,\qquad \tilde k_q = R(\theta_q)k_q.

Now expand the dot product one step at a time:

\tilde q_p^\top \tilde k_q = (R(\theta_p)q_p)^\top R(\theta_q)k_q = q_p^\top R(\theta_p)^\top R(\theta_q)k_q = q_p^\top R(\theta_q-\theta_p)k_q.

That last equality is the bridge: $R(\theta_p)^\top R(\theta_q)=R(\theta_q-\theta_p)$ . The score can therefore depend on relative position through the phase gap.

In practice, RoPE applies this to many 2D pairs with different frequencies. A common choice is:

\theta_{p,i} = p\,\omega_i, \qquad \omega_i = \mathrm{base}^{-2i/d},

where $d$ is head dimension and $i$ indexes the 2D pairs.

For one frequency, the phase gap is:

\theta_{q,i}-\theta_{p,i} = (q-p)\omega_i.

RoPE applies this rotation independently to each 2D coordinate pair $(2i,2i+1)$ of a head, so $d$ is typically even and $i \in \{0,\dots,\frac d2-1\}$ . Multiple frequencies let different coordinate pairs act like clocks with different tick rates.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Code witness 1import numpy as np def R(theta): c, s = np.cos(theta), np.sin(theta) return np.array([[c, -s]...python

import numpy as np

def R(theta):
    c, s = np.cos(theta), np.sin(theta)
    return np.array([[c, -s], [s, c]])

def rope_dot_direct(q, k, p, qpos, w):
    return float((R(p * w) @ q) @ (R(qpos * w) @ k))

def rope_dot_relative(q, k, p, qpos, w):
    return float(q @ (R((qpos - p) * w) @ k))

q = np.array([1.0, 0.2])
k = np.array([0.3, 1.0])
w = 0.7  # one frequency, for illustration

for delta in [0, 1, 2, 4, 8]:
    direct = rope_dot_direct(q, k, p=5, qpos=5 + delta, w=w)
    relative = rope_dot_relative(q, k, p=5, qpos=5 + delta, w=w)
    shifted = rope_dot_direct(q, k, p=0, qpos=delta, w=w)
    assert np.allclose(direct, relative)
    print("delta =", delta, "dot =", round(direct, 3), "shifted =", round(shifted, 3))

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the demo to predict the invariant before the rotating-vector lab appears. In the toy setting, the same base query/key vectors are compared at shifted positions, so preserving the relative phase preserves the positional part of the score.

Keep the scope narrow: RoPE is the rotary query-key phase mechanism. Long-context scaling methods such as position interpolation, YaRN, or LongRoPE change how positions map to phases; KV-cache methods change serving memory. Those are follow-on repairs, not the basic RoPE claim.

Live Concept Demo

Explore Rotary Position Embeddings (RoPE)

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 3/5undergraduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Rotary Position Embeddings (RoPE) should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

A positional encoding that rotates queries and keys so attention depends on relative position via phase differences.

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Rotary Position Embeddings (RoPE) should make visible.

Visual Inquiry

Make the image answer a mathematical question

A positional encoding that rotates queries and keys so attention depends on relative position via phase differences.

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Rotary Position Embeddings (RoPE) easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

paper · 2021RoFormer: Enhanced Transformer with Rotary Position EmbeddingSu et al.

Primary RoPE source. Sections 3.1-3.2 derive position-dependent rotations for q/k and show the query-key inner product uses the relative rotary product R_{Theta,n-m}.

Open source

Claim Review

A positional encoding that rotates queries and keys so attention depends on relative position via phase differences.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources1 reference

su-2021-roformer

Witnesses4 local objects

Use equation, code, and demo objects to check whether the source support is operational.

Substantively reviewedRoPE encodes absolute token positions by rotating query and key vectors, so their dot product can depend on relative position through the phase difference between positions.Claim metadata: source checked

Su et al. state that RoPE encodes absolute position with a rotation matrix and incorporates explicit relative-position dependency in self-attention; Sec. 3.1 frames the q-k inner product as a function of embeddings plus m-n, and Sec. 3.2 derives q/k rotations whose dot product contains R_{Theta,n-m}.

Sources: RoFormer: Enhanced Transformer with Rotary Position EmbeddingChecks only RoPE's rotary q/k attention-score mechanism; not RoPE scaling, arbitrary long-context extrapolation, YaRN/LongRoPE, KV-cache behavior, or production model performance.A bounded review summary is present; still check caveats and exact source scope.

Checked RoFormer abstract/introduction and Sec. 3.1-3.2: RoPE uses position-dependent rotations for q/k, the 2D complex form has phase gap m-n, and the general self-attention score contains R_{Theta,n-m}. Local math/code/demo witness the toy relative-angle mechanism.

Reviewer: codex+oracle; reviewed 2026-05-07

source-span-su-2021-roformer math-object-1 math-object-2 code-witness-1 interactive-demo

Source support candidates

paper 2021RoFormer: Enhanced Transformer with Rotary Position Embedding

Primary RoPE source. Sections 3.1-3.2 derive position-dependent rotations for q/k and show the query-key inner product uses the relative rotary product R_{Theta,n-m}.

Mechanism witnesses

Equation 1

R(\theta) = \begin{pmatrix} \cos\theta & -\sin\theta \\\\ \sin\theta & \cos\theta \end{pmatrix}.

Equation 2

\tilde q_p = R(\theta_p)q_p,\qquad \tilde k_q = R(\theta_q)k_q.

Code witness 1import numpy as np def R(theta): c, s = np.cos(theta), np.sin(theta) return np.array([[c, -s]...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

A positional encoding that rotates queries and keys so attention depends on relative position via phase differences.

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in Rotary Position Embeddings (RoPE).

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Object research drawerClose

ConceptRotary Position Embeddings (RoPE)Attention & Transformers

Code witness comparisonRotary Position Embeddings (RoPE) code witness 1return float(q @ (R((qpos - p) * w) @ k))Prediction before revealRotary Position Embeddings (RoPE) interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes Rotary Position Embeddings (RoPE) click without losing the math?Local snapshot ready

Research Room

Attach the question to an exact object

Pick the concept, equation, source, code witness, claim, misconception, or demo state before asking for help. The handoff stays grounded to that object.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptAttention & Transformers

Rotary Position Embeddings (RoPE)

Anchored question

What is the smallest example that makes Rotary Position Embeddings (RoPE) click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays locally in this browser for concept:attention-transformers/rope.

Draft noteNext action

No local draft saved.

Evidence to inspect

Source ids to inspect: su-2021-roformer
Definition, prerequisite, and contrast concept links
The equation or code witness that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Rotary Position Embeddings (RoPE) Object key: concept:attention-transformers/rope Context: Attention & Transformers Anchor id: concept/concept-notebook/attention-transformers/rope Open question: What is the smallest example that makes Rotary Position Embeddings (RoPE) click without losing the math? Evidence to inspect: - Source ids to inspect: su-2021-roformer - Definition, prerequisite, and contrast concept links - The equation or code witness that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

Open source object

concept/concept-notebook/attention-transformers/rope
concept:attention-transformers/rope

Learning Map

Before / Now / Try / Next

BeforeScaled Dot-Product Attention & Transformer Layers

NowIntuition → Math → Code → Demo

TryManipulate one control and predict the visible change.

NextEfficient Attention at Scale: KV Cache, GQA & FlashAttention

Intuitionready
Mathready
Codeready
Interactive Demoready

Object Companion

Ask beside the selected object

A positional encoding that rotates queries and keys so attention depends on relative position via phase differences.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Attention & Transformers concept. Learning surface: Rotary Position Embeddings (RoPE). What this page says: A positional encoding that rotates queries and keys so attention depends on relative position via phase differences. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

Attention & Transformers

attention-transformersropepositional-encodinglong-context

Prerequisites

Scaled Dot-Product Attention & Transformer Layers

Leads To

Efficient Attention at Scale: KV Cache, GQA & FlashAttention Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization

Tokenization & Vocabulary Design Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization Positional Encoding

Within this domain