Begin with attention

What will this token copy?

Follow one query: match keys, mix values, then change an input and explain an unfamiliar result.

Start the attention lesson

This tab carries your example once; new tabs and reloads start separately.

Change the query · one unmasked row

Shared normalization: m = 1.414; Z = 1.736.

  1. Source Aindex 0
    Dot
    0
    Scaled
    0
    Share
    0.14
    Carry w_Av_A
    [0.28, 0]
  2. Source Bindex 1
    Dot
    2
    Scaled
    1.414
    Share
    0.576
    Carry w_Bv_B
    [0, 1.152]
  3. Source Cindex 2
    Dot
    1
    Scaled
    0.707
    Share
    0.284
    Carry w_Cv_C
    [-0.284, -0.284]

Output o = [-0.004, 0.868]

Compared with q = [2, 1]:

Scores same; shares same; output same.

Worked example, not a test or model run. Given vectors, not learned meanings. Match the query with keys; mix values, not keys. All three sources are allowed in this view.

1 · Match and scaleq · kᵢ → sᵢ = (q · kᵢ) / √2sum(qj * kj for qj, kj in zip(q, k)) / sqrt(2)
2 · Share one denominatorwᵢ = exp(sᵢ − m) / Zexps = [exp(s - max(scores)) for s in scores]weights = [e / sum(exps) for e in exps]
3 · Carry the valuescᵢ = wᵢvᵢ; o = Σᵢ cᵢsum(w * v[j] for w, v in zip(weights, values))

m is the shared maximum scaled score; Z = exp(s_A − m) + exp(s_B − m) + exp(s_C − m). Subtracting the same maximum avoids large exponentials without changing the shares. Each carry is a weighted source value; sum the three contributions coordinate by coordinate for output o.

The reference uses the same current keys and values, with q = [2, 1]. Not every query edit changes the mixture. Solid amber borders mark numerical changes; dashed teal borders mark values unchanged within 1e-12. Labels round to three decimals, not the calculation.

Source A · index 0
k_A = [-1, 2]; v_A = [2, 0]
Source B · index 1
k_B = [1, 0]; v_B = [0, 2]
Source C · index 2
k_C = [0, 1]; v_C = [-1, -1]
Inspect precise numbers and input coordinates

Finite JavaScript numbers before display rounding; −0 is retained where supplied. Tiny allowed shares can underflow to zero; that is not a mask.

q = [2, 1]
A: key=[-1, 2]; value=[2, 0]; dot=0; score=0; share=0.14002924504337802; contribution=[0.28005849008675604, 0]
B: key=[1, 0]; value=[0, 2]; dot=2; score=1.414213562373095; share=0.575975345215362; contribution=[0, 1.151950690430724]
C: key=[0, 1]; value=[-1, -1]; dot=1; score=0.7071067811865475; share=0.28399540974126003; contribution=[-0.28399540974126003, -0.28399540974126003]
m=1.414213562373095; stable denominator=1.736185425829454
output=[-0.00393691965450399, 0.8679552806894639]
Read the full Python witness

This runnable Python witness uses the current controls and the same labels. No NumPy or random inputs are needed.

from math import exp, sqrt

labels = ["A","B","C"]
q = [2,1]
keys = [[-1,2],[1,0],[0,1]]
values = [[2,0],[0,2],[-1,-1]]

d_k = len(q)  # key dimension, NOT number of sources or value width
raw_scores = [sum(qj * kj for qj, kj in zip(q, k)) for k in keys]
scores = [s / sqrt(d_k) for s in raw_scores]
exps = [exp(s - max(scores)) for s in scores]
weights = [e / sum(exps) for e in exps]
output = [sum(w * v[j] for w, v in zip(weights, values))
          for j in range(len(values[0]))]

for label, raw, score, weight in zip(labels, raw_scores, scores, weights):
    print(label, round(raw, 3), round(score, 3), round(weight, 3))
print("output", [round(x, 3) for x in output])

Expected output: [-0.004, 0.868] (rounded to three decimals).

Inspect key and value geometry
Key space: query and source keysQuery q is [2, 1]. Source coordinates and calculations are available in the labelled numeric readouts.xy0k_Ak_Bk_Cq
Key space only. Each grid interval is 1 coordinate unit on both axes. Amber marks the query q you control; ink marks the three fixed keys.
Value space: weighted output and source valuesOutput o is [-0.004, 0.868]. Source coordinates and calculations are available in the labelled numeric readouts.xy0v_Av_Bv_Co
Value space only. Each grid interval is 1 coordinate unit on both axes. Teal marks the mixed output o; the triangle bounds its possible positions.

Stay with attention

One mechanism. Four ways to inspect it.

All four links stay in the attention notebook. The guided example is unmasked; the math and code also introduce causal masking.

Go deeper · Transformer systems

What gets smaller when four query heads share one memory?

An advanced KV-memory example. Predict the memory effect before revealing the calculation; it is not a measured speedup or a model-quality result.

Inspect the advanced example preview

Attention head sharing

One mechanism, still sealed

Result sealed

Thirty-two query heads enter a sealed head-sharing comparison. The changed quantity and calculated result are not shown.

  1. Head diagram
  2. Equation
  3. Code
  4. Calculated result

The evaluated count and memory stay sealed until you lock in a guess or choose Just show me.

Test the KV-memory example

Saved in this browser only if you lock in a prediction.

Topic starting points

Follow your next question.

See all learning paths

51 published notebooks

The atlas, when you need more context.

Open the Atlas

Depth varies by topic. Selected notebooks include mathematics, code, or interactive figures. Published counts include notebooks with prerequisite gaps; curated starting links above do not.

Beyond the first lesson

Test an idea. Frame a question. Read evidence.

Make a prediction

Deterministic investigations take your answer before showing the calculation. No score, streak, or account; selected predictions can be saved in this browser.

Enter the Lab

Frame an open question

Research offers browser-local drafts, not hosted experiments or a live agent. The current objective is conditional, with model-free feasibility work only; earlier protocol history remains parked. No study is running and no scientific evidence or result exists.

Browse research questions

Interrogate submitted evidence

Verify reads the scores and declarations you provide. It does not observe runs, verify declarations, execute models, or issue durable receipts.

Open Verify

Continuous Function is an independent open research project. These surfaces do not establish mastery, learning effects, or an end-to-end research runtime.