Attention — Continuous Function

Why It Matters for Modern Models

GPT-4, Claude, Gemini, Llama: giant stacks of decoder-only transformer blocks with causal self-attention
Stable Diffusion: U-Net with self- and cross-attention between image latents and text embeddings
Sora: diffusion transformer operating on spacetime patches (video tokens)

What Tutorials Skip

What is still poorly explained in textbooks and papers:

Geometric picture of Q–K dot products as measuring angles between feature directions, and how softmax turns those into a distribution of "who to copy from"
How multi-head attention effectively builds a set of learned kernels over positions/features, and why this is strictly more flexible than fixed kernels

Interactive Visualization

Row 0 (A) · unmasked projection.

One query · three sources · one continuous example

Follow one query through attention

Change the query and follow the same A, B, and C into a mixture. Move at your own pace; no prediction gate is required.

Worked instrument, not a test. These given vectors illustrate one unmasked attention head; they are not learned word meanings.

Change the query · one unmasked rowQuery q, x: 2Query q, y: 1

Shared normalization: m = 1.414; Z = 1.736.

Source Aindex 0
Dot
0
Scaled
0
Share
0.14
Carry w_Av_A
[0.28, 0]
Source Bindex 1
Dot
2
Scaled
1.414
Share
0.576
Carry w_Bv_B
[0, 1.152]
Source Cindex 2
Dot
1
Scaled
0.707
Share
0.284
Carry w_Cv_C
[-0.284, -0.284]

Output o = [-0.004, 0.868]

Compared with q = [2, 1]:

Scores same; shares same; output same.

Worked example, not a test or model run. Given vectors, not learned meanings. Match the query with keys; mix values, not keys. All three sources are allowed in this view.

1 · Match and scaleq · kᵢ → sᵢ = (q · kᵢ) / √2sum(qj * kj for qj, kj in zip(q, k)) / sqrt(2)

2 · Share one denominatorwᵢ = exp(sᵢ − m) / Zexps = [exp(s - max(scores)) for s in scores]weights = [e / sum(exps) for e in exps]

3 · Carry the valuescᵢ = wᵢvᵢ; o = Σᵢ cᵢsum(w * v[j] for w, v in zip(weights, values))

m is the shared maximum scaled score; Z = exp(s_A − m) + exp(s_B − m) + exp(s_C − m). Subtracting the same maximum avoids large exponentials without changing the shares. Each carry is a weighted source value; sum the three contributions coordinate by coordinate for output o.

The reference uses the same current keys and values, with q = [2, 1]. Not every query edit changes the mixture. Solid amber borders mark numerical changes; dashed teal borders mark values unchanged within 1e-12. Labels round to three decimals, not the calculation.

Source A · index 0: k_A = [-1, 2]; v_A = [2, 0]
Source B · index 1: k_B = [1, 0]; v_B = [0, 2]
Source C · index 2: k_C = [0, 1]; v_C = [-1, -1]

Inspect precise numbers and input coordinates

Finite JavaScript numbers before display rounding; −0 is retained where supplied. Tiny allowed shares can underflow to zero; that is not a mask.

q = [2, 1]
A: key=[-1, 2]; value=[2, 0]; dot=0; score=0; share=0.14002924504337802; contribution=[0.28005849008675604, 0]
B: key=[1, 0]; value=[0, 2]; dot=2; score=1.414213562373095; share=0.575975345215362; contribution=[0, 1.151950690430724]
C: key=[0, 1]; value=[-1, -1]; dot=1; score=0.7071067811865475; share=0.28399540974126003; contribution=[-0.28399540974126003, -0.28399540974126003]
m=1.414213562373095; stable denominator=1.736185425829454
output=[-0.00393691965450399, 0.8679552806894639]

Read the full Python witness

This runnable Python witness uses the current controls and the same labels. No NumPy or random inputs are needed.

from math import exp, sqrt

labels = ["A","B","C"]
q = [2,1]
keys = [[-1,2],[1,0],[0,1]]
values = [[2,0],[0,2],[-1,-1]]

d_k = len(q)  # key dimension, NOT number of sources or value width
raw_scores = [sum(qj * kj for qj, kj in zip(q, k)) for k in keys]
scores = [s / sqrt(d_k) for s in raw_scores]
exps = [exp(s - max(scores)) for s in scores]
weights = [e / sum(exps) for e in exps]
output = [sum(w * v[j] for w, v in zip(weights, values))
          for j in range(len(values[0]))]

for label, raw, score, weight in zip(labels, raw_scores, scores, weights):
    print(label, round(raw, 3), round(score, 3), round(weight, 3))
print("output", [round(x, 3) for x in output])

Expected output: [-0.004, 0.868] (rounded to three decimals).

Inspect key and value geometry

Key space only. Each grid interval is 1 coordinate unit on both axes. Amber marks the query q you control; ink marks the three fixed keys.

Value space only. Each grid interval is 1 coordinate unit on both axes. Teal marks the mixed output o; the triangle bounds its possible positions.

Step 1 of 4 · Vectors

What does this query have in common with each key?

A, B, and C name the same three sources throughout. Compare q with their keys in key space. Their values are payloads in a different space, not arrows to compare with q.

These are optional inspection controls. The connected scores, shares and mixture stay visible above.

Read the current numbers

Shapes: q and each k have d_k = 2 coordinates. Each v and the output have d_v = 2 coordinates. Equal widths here do not make the spaces identical.

q = [q₁, q₂]; k_A = [−1, 2]; k_B = [1, 0]; k_C = [0, 1]

i names a source; j runs over A, B, C. s is a scaled score, w its share, and o the output. Displayed decimals are rounded; calculations are not.

Keys in key space
Source	Key k	q
A	[-1, 2]	[2, 1]
B	[1, 0]	[2, 1]
C	[0, 1]	[2, 1]

Reset example restores the query and first value’s x coordinate only. All other held inputs stay unchanged.

All sources are allowed here; this is not a masked sequence output. Query edits return to row 0; value edits are shared. Other rows and the held mask stay unchanged.

Practice, help and export

A study detour counts as consultation for the attempt you leave. Clearing a response or resetting values keeps its help history.

Held example · 3 queries · 3 sources. Started from the A/B/C sequence.

Study history for this visit

These are help exposures, not completion or learning scores. Individual attempts retain their own feedback and help. Reloading the page clears this local history; it does not establish a fresh unaided assessment.

Guided attention notebook · worked study · sequence-abc-v1

Core Math (Optional Deep Dive)

If you want intuition first, start with the key equation and local visualization. Come back here for the full walkthrough.

Key Equation

\text{Attn}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

Single attention head:

\text{Attn}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

where $Q = XW_Q,\ K = XW_K,\ V = XW_V$ . Multi-head attention concatenates several such heads.

A standard transformer block:

\begin{aligned} H' &= \mathrm{MHA}(\mathrm{LN}(H)) + H \\ H^{\text{out}} &= \mathrm{MLP}(\mathrm{LN}(H')) + H' \end{aligned}

Canonical Papers

Attention Is All You Need

Vaswani et al.2017NeurIPS

Read paper →

Connections

Next Moves

Choose the next question to carry this object forward.

Semantic Connections

Reuse / Same Technique

🔀Softmax routing→MoE ∂Cross-attention→Diffusion ◎Dot-product logits→Embeddings 🎲Softmax selection→Decoding 🖼️Cross-attention fusion→Multimodal VLP ⊛Copy via attention→Circuits 🖼️Cross-attention fusion→Multimodal VLP

Break / Breaks When

⚙O(n²) memory→Efficient Attention 📏Attention dilution→Long Context

Fix / Invented to Fix

↻Position encoding→RoPE ⚙KV cache memory→Efficient Attention ⏩Autoregressive latency→Speculative Decoding ⚡Production serving→LLM Serving 🔀Linear complexity→SSMs & Hybrids ⚙O(n²) memory→Efficient Attention

Dual / Mathematical Dual

🔀Kernel ↔ Recurrence→SSMs & Hybrids

Map / Analogy

🔀Sparse selection→MoE

Scaled Dot-Product Attention & Transformer Layers

Keep the equation fixed; move through the evidence.