Browse linear algebra
Build the mathematical language needed for modern model training.
Begin with attention
Follow one query: match keys, mix values, then change an input and explain an unfamiliar result.
This tab carries your example once; new tabs and reloads start separately.
Shared normalization: m = 1.414; Z = 1.736.
Output o = [-0.004, 0.868]
Compared with q = [2, 1]:
Scores same; shares same; output same.
Worked example, not a test or model run. Given vectors, not learned meanings. Match the query with keys; mix values, not keys. All three sources are allowed in this view.
sum(qj * kj for qj, kj in zip(q, k)) / sqrt(2)exps = [exp(s - max(scores)) for s in scores]weights = [e / sum(exps) for e in exps]sum(w * v[j] for w, v in zip(weights, values))m is the shared maximum scaled score; Z = exp(s_A − m) + exp(s_B − m) + exp(s_C − m). Subtracting the same maximum avoids large exponentials without changing the shares. Each carry is a weighted source value; sum the three contributions coordinate by coordinate for output o.
The reference uses the same current keys and values, with q = [2, 1]. Not every query edit changes the mixture. Solid amber borders mark numerical changes; dashed teal borders mark values unchanged within 1e-12. Labels round to three decimals, not the calculation.
Finite JavaScript numbers before display rounding; −0 is retained where supplied. Tiny allowed shares can underflow to zero; that is not a mask.
q = [2, 1] A: key=[-1, 2]; value=[2, 0]; dot=0; score=0; share=0.14002924504337802; contribution=[0.28005849008675604, 0] B: key=[1, 0]; value=[0, 2]; dot=2; score=1.414213562373095; share=0.575975345215362; contribution=[0, 1.151950690430724] C: key=[0, 1]; value=[-1, -1]; dot=1; score=0.7071067811865475; share=0.28399540974126003; contribution=[-0.28399540974126003, -0.28399540974126003] m=1.414213562373095; stable denominator=1.736185425829454 output=[-0.00393691965450399, 0.8679552806894639]
This runnable Python witness uses the current controls and the same labels. No NumPy or random inputs are needed.
from math import exp, sqrt
labels = ["A","B","C"]
q = [2,1]
keys = [[-1,2],[1,0],[0,1]]
values = [[2,0],[0,2],[-1,-1]]
d_k = len(q) # key dimension, NOT number of sources or value width
raw_scores = [sum(qj * kj for qj, kj in zip(q, k)) for k in keys]
scores = [s / sqrt(d_k) for s in raw_scores]
exps = [exp(s - max(scores)) for s in scores]
weights = [e / sum(exps) for e in exps]
output = [sum(w * v[j] for w, v in zip(weights, values))
for j in range(len(values[0]))]
for label, raw, score, weight in zip(labels, raw_scores, scores, weights):
print(label, round(raw, 3), round(score, 3), round(weight, 3))
print("output", [round(x, 3) for x in output])
Expected output: [-0.004, 0.868] (rounded to three decimals).
Stay with attention
All four links stay in the attention notebook. The guided example is unmasked; the math and code also introduce causal masking.
Go deeper · Transformer systems
An advanced KV-memory example. Predict the memory effect before revealing the calculation; it is not a measured speedup or a model-quality result.
Attention head sharing
Thirty-two query heads enter a sealed head-sharing comparison. The changed quantity and calculated result are not shown.
The evaluated count and memory stay sealed until you lock in a guess or choose Just show me.
Saved in this browser only if you lock in a prediction.
Topic starting points
Build the mathematical language needed for modern model training.
Move from attention mechanics to long-context inference decisions.
Use the foundations to inspect generation, alignment, and representations.
51 published notebooks
Depth varies by topic. Selected notebooks include mathematics, code, or interactive figures. Published counts include notebooks with prerequisite gaps; curated starting links above do not.
Beyond the first lesson
Deterministic investigations take your answer before showing the calculation. No score, streak, or account; selected predictions can be saved in this browser.
Enter the LabResearch offers browser-local drafts, not hosted experiments or a live agent. The current objective is conditional, with model-free feasibility work only; earlier protocol history remains parked. No study is running and no scientific evidence or result exists.
Browse research questionsVerify reads the scores and declarations you provide. It does not observe runs, verify declarations, execute models, or issue durable receipts.
Open VerifyContinuous Function is an independent open research project. These surfaces do not establish mastery, learning effects, or an end-to-end research runtime.