Representation learning maps inputs to vectors whose distances or inner products reflect relations in the data. Token embeddings in language models and the joint image–text embeddings of CLIP are examples; CLIP trains its encoders with a contrastive loss on matching and non-matching pairs (Radford et al., 2021).
A newer notebook covers this concept. This older page keeps its summary, math and sources.
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Before you move a control in the demo, say what you expect to change.
CheckCheck against the math and the papers.
Compare what the demo shows with the key equation and with the papers cited below.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueContinue in the newer notebook.
The newer notebook covers the same concept and has an interactive demo.
Why It Matters for Modern Models
Representation learning maps inputs to vectors whose distances or inner products reflect relations in the data. Token embeddings in language models and the joint image–text embeddings of CLIP are examples; CLIP trains its encoders with a contrastive loss on matching and non-matching pairs (Radford et al., 2021).
Retrieval and recommendation systems search by nearest neighbors in an embedding space, so the geometry of that space decides what counts as similar.
Questions to Think About
How do representations change from layer to layer, from local features to more global ones?
Interactive Demos
Core Math
If you prefer intuition first, try the demos above, then come back for the math.
The sum runs over the matching ("positive") y and N−1 non-matching ("negative") candidates sampled independently from the marginal distribution of y. The loss pulls positive pairs together and pushes negatives apart. Minimizing it maximizes a lower bound on the mutual information between the two views, I≥logN−L (van den Oord et al., 2018); that bound alone does not explain why the learned representations are useful (Tschannen et al., 2020).
Key Papers
Representation Learning: A Review and New Perspectives
Temperature scaling: the same technique appears in Decoding & Sampling. Open Decoding & Sampling
Contrastive learning uses temperature τ to control softmax sharpness; decoding temperature does the same to token probabilities. In both places, temperature sets the entropy of a softmax: low values sharpen it, high values flatten it.
InfoNCE similarity matrix: the same technique appears in Multimodal Pretraining. Open Multimodal Pretraining
CLIP training is contrastive representation learning across two modalities: build the matrix of image–text similarities in a batch and apply a softmax cross-entropy along each row and each column (Radford et al., 2021). SigLIP keeps the similarity matrix but replaces the softmax with an independent sigmoid loss on every pair (Zhai et al., 2023).
Dot-product logits: the same technique appears in Attention. Open Attention
Attention forms scores from query–key dot products divided by √d_k; contrastive learning forms them from embedding dot products divided by a temperature τ. Both build a matrix of dot-product similarities and apply a softmax to each row. Attention uses the resulting weights to average value vectors, while contrastive learning treats them as predicted probabilities and trains them with cross-entropy.
Algorithmic feature: an analogy between Embeddings and Circuits. Open Circuits
Induction heads are "algorithmic features"—circuits that implement pattern-completion algorithms. Understanding them bridges circuit-level interpretability and representation-level analysis.
Dictionary learning: the same technique appears in Sparse Autoencoders. Open Sparse Autoencoders
SAEs apply dictionary learning to a network's activations: each activation is explained as a sparse combination of learned directions, which are often easier to interpret than individual neurons. The dictionary has more directions than the activations have dimensions, so it is not a basis.
Both rely on many concepts being represented as directions in activation space. Steering adds such a direction, for example a "refusal direction", to the activations (Arditi et al., 2024): the same geometry, used to change behaviour instead of reading it.
Shaping behaviour: the same technique appears in KTO. Open KTO
Both move a model's behaviour towards what people prefer, by different routes: KTO trains the weights on outputs labelled desirable or undesirable, while representation steering leaves the weights alone and adds a direction to the hidden states at inference time. Whether training of this kind works by moving activations along such directions is an open question.
Which ideas are its mathematical duals?
Bottleneck viewpoint: Learning Theory is its mathematical dual. Open Learning Theory
Representation learning is often "keep what matters, drop what doesn't." The information bottleneck formalizes that as maximizing I(Z;Y) while penalizing I(Z;X)—a dual language for the same design tradeoff.