∇Continuous Function

Transformer Systems Lab

RoPE phase station

Carry the query-key route from the first gradient lab into positional geometry: predict what equal shifts preserve, then test a key-only transfer before moving to KV memory.

station 04local LabCase v1prediction firstKV handoff
Start RoPE StationBack to Text UpdateOpen Route Cockpit
Carried objectsame query-key route, now with position(R_p q)^T (R_q k) = q^T R(theta_q - theta_p) k
Baselinep=2, q=0phase gap fixed
Equal shiftp+3, q+3dot score holds
Key onlyq+1transfer changes
01Object

one rotated Q/K pair

02Gap

absolute position vs phase gap

03Prediction

commit equal-shift claim

04Manipulation

move p, q, shift, frequency

05Evidence

phase table and dot score

06Invariant

relative phase controls score

07Transfer

key-only perturbation

08Next

KV memory handoff

Carry-in contract

The same attention route now asks what position changes.

RoPE is not a new destination or a long-context guarantee. It is the next operable object in the same Transformer Systems Lab case: the query-key score now contains rotation and relative phase.

Carried objectOne query-key score from the text-gradient route.
PredictionWill an equal shift change the dot score?
EvidenceBaseline, equal-shift, and key-only phase witnesses.
Next moveWhich positioned keys and raw values must be cached?

Operable Station 03

RoPE: carry the query-key route into phase

Keep the same query/key attention object from the route. Now ask which part of the score survives when position changes.

CF.LABCASE.V1Prediction neededNo prediction yet is preserved as the first answer.
Mechanism sourceRoFormer / RoPERotary position embedding as relative phase in Q/K pairs.Route sourceAttention Is All You NeedRoPE changes the query-key score geometry, not the value-copy rule.

Selected Object

One 2D query/key frequency pair

(R_p q)^T (R_q k) = q^T R(theta_q - theta_p) k

Use one tiny coordinate pair so the cancellation is inspectable. The content vectors stay fixed; only the positions and frequency band move.

q[0.72, 0.32]k[0.58, 0.45]omega0.42
Open RoPE notebook object

Prediction Checkpoint

Shift both positions by +3: q 2 -> 5, k 0 -> 3. What should happen to the query-key score?

First answerNo prediction yetRevisions never overwrite the first answer.

Manipulation

Move positions, then re-predict.

query position p
key position q
equal shift
frequency band

Default witness band: enough movement to see the counterfactual.

Route spine

This station is finished only when it hands a usable invariant forward.

The route is deliberately narrow here: finish the phase invariant, preserve the first answer, save the evidence, then move one step into KV memory.

  1. 01Text to one update

    Toy text, causal route, p-y loss signal

    ready
  2. 02RoPE phase

    Rotated query/key pair and relative phase

    active
  3. 04Long-context use

    Position range, retrieval quality, and memory pressure

    later