10Representations

◎Representation Learning & Embedding Geometry

Canonical Papers

Bengio et al.2013IEEE TPAMI

Learn a mapping $f_\theta: \mathcal X \to \mathbb R^d$ such that inner products or distances reflect meaningful relations.

Contrastive objective (InfoNCE-style):

\mathcal L = - \mathbb E \left[ \log \frac{\exp(\mathrm{sim}(f(x),g(y))/\tau)}{\sum_{y'} \exp(\mathrm{sim}(f(x), g(y'))/\tau)} \right]

This pushes "positive" pairs together, "negatives" apart; at optimum, it maximizes a lower bound on mutual information between views.

Key Equation

\mathcal L = - \mathbb E \left[ \log \frac{\exp(\mathrm{sim}(f(x),g(y))/\tau)}{\sum_{y'} \exp(\mathrm{sim}(f(x), g(y'))/\tau)} \right]

Word & token embeddings in LMs, vision embeddings in CLIP-like models, multimodal embeddings in Gemini and GPT-4V
Latent spaces of Stable Diffusion designed so distances correspond to semantic similarity

What is still poorly explained in textbooks and papers:

Geometric explanation of anisotropy (representations bunch along a few directions) and how normalization/whitening alter behavior
Visuals showing how representations evolve across layers (local to global features)

Explore this concept from different angles — like a mathematician would.

🔄 Same Technique

↔️ Mathematical Dual

≈ Analogy