Positional Encoding

The classic sinusoidal encoding (Vaswani et al., 2017, §3.5) defines, for model dimension $d$ and position $m$, with base 10,000:

with frequencies

Two useful facts.

Shift structure. The dot product between the encodings of positions $m$ and $n$ depends only on their offset:

Each term comes from one sine–cosine pair and the cosine difference identity, $\sin(m\omega)\sin(n\omega)+\cos(m\omega)\cos(n\omega)=\cos\bigl((m-n)\omega\bigr)$, so moving both positions by the same amount leaves the dot product unchanged. This is a statement about the position vectors alone. Once they are added to token embeddings and passed through learned query and key projections, an attention score also depends on the tokens.

Additive absolute encoding. The simplest way to use PE is to add it to token embeddings:

Relative-position methods (like RoPE) instead change the attention computation so that the score between a query at position $m$ and a key at position $n$ depends on the offset $n-m$ directly.

One pair, worked by hand

Take a single pair $s(m)=\bigl(\sin(m\omega),\cos(m\omega)\bigr)$ and set $\omega=\pi/2$, a quarter turn per position. This frequency is chosen so the arithmetic is exact; it is not one of the published frequencies. Every $s(m)$ is then one of four points on the unit circle:

Positions $0$ and $1$ give $s(0)\cdot s(1)=0$. Moving both two places later gives $s(2)\cdot s(3)=0$ again: the offset is still $-1$. Now keep $m=0$ and move $n$ to $5$. Four quarter turns make a full turn, so $s(5)=s(1)$ and $s(0)\cdot s(5)=0$. This pair cannot tell distance $1$ from distance $5$: one frequency repeats, so its dot product is not a measure that falls as positions move apart.

The collision belongs to the single pair. The full encoding stacks pairs that turn at different speeds, and the slow ones have not come round yet: with $d=8$, $\mathrm{PE}(0)\cdot\mathrm{PE}(1)\approx 3.535$ but $\mathrm{PE}(0)\cdot\mathrm{PE}(5)\approx 3.160$. The demo works these numbers one step at a time.