Foundation Lab

Scaled Dot-Product Attention & Transformer Layers

GPT-4, Claude, Gemini, Llama: giant stacks of decoder-only transformer blocks with causal self-attention

Concept 2 of 100Core TrainingPhase 1

Open New Notebook All Foundations

#2AttentionCore Training

key equation\text{Attn}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

Selected Foundation Object

Keep the equation fixed; move through the evidence.

Concept 2 of 100AttentionCore Training / Phase 1: Core probabilistic training + transformers

Current question

Geometric picture of Q–K dot products as measuring angles between feature directions, and how softmax turns those into a distribution of "who to copy from"

\text{Attn}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

PredictionCommit before the demo.

Ask what should change when the equation is manipulated, then let the visualization test that expectation.

EvidenceCompare local witness and source.

Use the runnable panel, the key equation, and canonical papers as separate forms of evidence for the same object.

InvariantName what survives notation changes.

The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.

Next moveContinue in the newer notebook.

This atlas page keeps the working demo; the domain notebook carries the fuller Intuition -> Math -> Code -> Demo sequence.

Open demo Inspect equation Open newer notebook Next foundation

Why It Matters for Modern Models

GPT-4, Claude, Gemini, Llama: giant stacks of decoder-only transformer blocks with causal self-attention
Stable Diffusion: U-Net with self- and cross-attention between image latents and text embeddings
Sora: diffusion transformer operating on spacetime patches (video tokens)

What Tutorials Skip

What is still poorly explained in textbooks and papers:

Geometric picture of Q–K dot products as measuring angles between feature directions, and how softmax turns those into a distribution of "who to copy from"
How multi-head attention effectively builds a set of learned kernels over positions/features, and why this is strictly more flexible than fixed kernels

Interactive Visualization

Core Math (Optional Deep Dive)

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation

\text{Attn}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

Single attention head:

\text{Attn}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

where $Q = XW_Q,\ K = XW_K,\ V = XW_V$ . Multi-head attention concatenates several such heads.

A standard transformer block:

\begin{aligned} H' &= \mathrm{MHA}(\mathrm{LN}(H)) + H \\ H^{\text{out}} &= \mathrm{MLP}(\mathrm{LN}(H')) + H' \end{aligned}

Canonical Papers

Attention Is All You Need

Vaswani et al.2017NeurIPS

Read paper →

Connections

Next Moves

Choose the next question to carry this object forward.

Semantic Connections

Reuse / Same Technique

🔀Softmax routing→MoE ∂Cross-attention→Diffusion ◎Dot-product logits→Embeddings 🎲Softmax selection→Decoding 🖼️Cross-attention fusion→Multimodal VLP ⊛Copy via attention→Circuits 🖼️Cross-attention fusion→Multimodal VLP

Break / Breaks When

⚙O(n²) memory→Efficient Attention 📏Attention dilution→Long Context

Fix / Invented to Fix

↻Position encoding→RoPE ⚙KV cache memory→Efficient Attention ⏩Autoregressive latency→Speculative Decoding ⚡Production serving→LLM Serving 🔀Linear complexity→SSMs & Hybrids ⚙O(n²) memory→Efficient Attention

Dual / Mathematical Dual

🔀Kernel ↔ Recurrence→SSMs & Hybrids

Map / Analogy

🔀Sparse selection→MoE