∇Continuous Function
2Core Training

⊗Scaled Dot-Product Attention & Transformer Layers

Canonical Papers

Attention Is All You Need

Vaswani et al.2017NeurIPS
Read paper →

Core Mathematics

Single attention head:

Attn(Q,K,V)=softmax ⁣(QK⊤dk)V\text{Attn}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

where Q=XWQ, K=XWK, V=XWVQ = XW_Q,\ K = XW_K,\ V = XW_V. Multi-head attention concatenates several such heads.

A standard transformer block:

H′=MHA(LN(H))+HHout=MLP(LN(H′))+H′\begin{aligned} H' &= \mathrm{MHA}(\mathrm{LN}(H)) + H \\ H^{\text{out}} &= \mathrm{MLP}(\mathrm{LN}(H')) + H' \end{aligned}
Key Equation
Attn(Q,K,V)=softmax ⁣(QK⊤dk)V\text{Attn}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

Interactive Visualization

Why It Matters for Modern Models

  • GPT-4, Claude, Gemini, Llama: giant stacks of decoder-only transformer blocks with causal self-attention
  • Stable Diffusion: U-Net with self- and cross-attention between image latents and text embeddings
  • Sora: diffusion transformer operating on spacetime patches (video tokens)

Missing Intuition

What is still poorly explained in textbooks and papers:

  • Geometric picture of Q–K dot products as measuring angles between feature directions, and how softmax turns those into a distribution of "who to copy from"
  • How multi-head attention effectively builds a set of learned kernels over positions/features, and why this is strictly more flexible than fixed kernels

Connections

Prerequisites

ℒML/CE/KL

Enables

◎Embeddings⊛Circuits↗Scaling⚙Efficient Attention↻RoPE⏩Speculative Decoding⚡LLM Serving🔀MoE⚡MoE Serving🔬Circuit Discovery🎚️Activation Steering📏Long Context🔀SSMs & Hybrids🖼️Multimodal VLP🎲Decoding

Next Moves

Explore this concept from different angles — like a mathematician would.

Semantic Connections

🔄 Same Technique
🔀Softmax routing→MoE∂Cross-attention→Diffusion◎Dot-product logits→Embeddings🎲Softmax selection→Decoding🖼️Cross-attention fusion→Multimodal VLP⊛Copy via attention→Circuits🖼️Cross-attention fusion→Multimodal VLP
⚠️ Breaks When
⚙O(n²) memory→Efficient Attention📏Attention dilution→Long Context
🔧 Invented to Fix
↻Position encoding→RoPE⚙KV cache memory→Efficient Attention⏩Autoregressive latency→Speculative Decoding⚡Production serving→LLM Serving🔀Linear complexity→SSMs & Hybrids⚙O(n²) memory→Efficient Attention
↔️ Mathematical Dual
🔀Kernel ↔ Recurrence→SSMs & Hybrids
≈ Analogy
🔀Sparse selection→MoE