Transformer Systems Lab
RoPE: what position does to an attention score
Rotary position embedding (RoPE) rotates each query and key by an angle proportional to its position. Predict whether shifting both positions by the same amount changes their score, then predict what happens when only the key moves.
What this step asks
The same query–key score, now with positions.
Step 1 scored a query against keys with a plain dot product. RoPE rotates the query and the key before that dot product, by angles set by their positions (Su et al., 2024). This step asks which part of the score still depends on position after the rotation.
Step 2 of 8
RoPE: rotate the query and key by position
The same kind of query–key score as step 1, now with positions. Which part of the score survives when both positions change?
Your answerPrediction neededYour first answer is kept even if you change it later.
Primary sourceSu et al. (2024). RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing 568.Introduces RoPE: queries and keys are rotated by position, so their dot product depends on the offset between positions.AttentionVaswani et al. (2017). Attention Is All You Need. NeurIPS.Scaled dot-product attention. RoPE changes how queries and keys are scored, not how values are mixed.
Example
One query and one key in two dimensions, at one frequency
(R_m q)^T (R_n k)R_m rotates the query q by m·ω, its position m times the frequency ω; R_n rotates the key k by n·ω. The vectors q and k stay fixed; only the positions and the frequency change, and each rotation is small enough to check by hand.
q[0.72, 0.32]k[0.58, 0.45]ω0.42
Open the RoPE notebookYour prediction
Shift both positions by +3: query 2 → 5, key 0 → 3. What happens to the query–key score?
First answerNo prediction yetChanging your answer later does not replace this one.
Controls
Change a setting, then predict again.
query position m
key position n
equal shift
frequency ω
Medium (default): 0.42 rad per position, enough movement to see each change clearly.
Outline of this lesson
Exampleone query–key score, now with position
(R_m q)^T (R_n k)one rotated query/key pair
absolute position or the gap between positions?
what an equal shift does to the score
move m, n, the shift and the frequency
phase table and dot-product score
shown after you answer
move only the key
KV memory
Where this step sits
From the score to the cache.
Finish the equal-shift and key-only cases, then save what you found. The next step uses it: the keys stored in a KV cache are the rotated keys from this step.