Selected relation: Embeddings to Attention: Comparison produces scores
Comparison produces scores
Authored explanation · Comparison
Embeddings supply coordinates. Attention compares a query with each key using a dot product, then divides by the square root of the key dimension. The result is one compatibility score per key, not yet a share of the output.
One concrete example
Let the query be [1] and keys A, B, C be [ln 2], [0], [0]. Their dot products are [ln 2, 0, 0]. Here the key dimension is 1, so division by its square root leaves those scores unchanged. ln 2 is approximately 0.693.
The same toy example across all three relations; no masking or dropout.
What stays true. For fixed queries and keys, changing a value does not change these scores.
Where this connection stops
A dot product depends on magnitudes as well as alignment. A large score is not proof that a position is relevant or correct.
Next investigation
In the attention notebook, choose Reset example, then Scores. Move Query q, x from 2 to 1; leave Query q, y at 1 and keep keys and values fixed. Which scores change, which stays the same, and how are the unchanged values mixed afterward?
Investigate AttentionPrerequisite repair and catalog links
Catalog prerequisites for Attention
Declared by the foundations catalog, not inferred from your selection. An arrow here reads prerequisite to concept. Open a repair only when you need it.
Catalog-declared related links (17)
These retain the catalog's direction and relation type. A related link is not a prerequisite or an additional authored explanation.
- Attention to MoE · same trick: Softmax routingOpen MoE
- Attention to Diffusion · same trick: Cross-attentionOpen Diffusion
- Attention to Efficient Attention · breaks when: O(n²) memoryOpen Efficient Attention
- Attention to RoPE · invented to fix: Position encodingOpen RoPE
- Attention to Efficient Attention · invented to fix: KV cache memoryOpen Efficient Attention
- MoE to Attention · analogy: Sparse selectionOpen MoE
- Attention to Embeddings · same trick: Dot-product logitsOpen Embeddings
- Attention to Decoding · same trick: Softmax selectionOpen Decoding
- Attention to SSMs & Hybrids · duality: Kernel ↔ RecurrenceOpen SSMs & Hybrids
- Attention to Long Context · breaks when: Attention dilutionOpen Long Context
- Attention to Speculative Decoding · invented to fix: Autoregressive latencyOpen Speculative Decoding
- Attention to LLM Serving · invented to fix: Production servingOpen LLM Serving
- Attention to Multimodal VLP · same trick: Cross-attention fusionOpen Multimodal VLP
- Circuits to Attention · same trick: Copy via attentionOpen Circuits
- SSMs & Hybrids to Attention · invented to fix: Linear complexityOpen SSMs & Hybrids
- Multimodal VLP to Attention · same trick: Cross-attention fusionOpen Multimodal VLP
- Efficient Attention to Attention · invented to fix: O(n²) memoryOpen Efficient Attention