Continuous Function

Foundation reference

Contrastive Learning & InfoNCE

Contrastive learning trains useful encoders without labels: a linear classifier on SimCLR features reached 76.5% top-1 accuracy on ImageNet, matching a supervised ResNet-50 (Chen et al., 2020).

Concept 49 of 100RepresentationsStudy phase 5
Go to the mathAll concepts
Key equation
L=−log⁡exp⁡(sim(zi,zj+)/τ)∑kexp⁡(sim(zi,zk)/τ)\mathcal{L} = -\log \frac{\exp(\text{sim}(z_i, z_j^+) / \tau)}{\sum_k \exp(\text{sim}(z_i, z_k) / \tau)}
Reading map and next steps

Study guide

Where to start

Concept 49 of 100Contrastive LearningRepresentations · Study phase 5: Representation & interpretability
Question

How does the temperature τ decide whether the loss concentrates on the hardest negatives or spreads its weight over all of them?

Go to the mathAll conceptsNext concept
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.

Pick a concrete input and say what the equation should give before you read the math.

CheckCheck against the math and the papers.

Work through the math, then read the cited papers for the original argument.

GeneralizeName what stays the same.

Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.

ContinueFollow the connections.

Use the prerequisites and related concepts at the end of the page to fill the next gap.

Why It Matters for Modern Models

  • Contrastive learning trains useful encoders without labels: a linear classifier on SimCLR features reached 76.5% top-1 accuracy on ImageNet, matching a supervised ResNet-50 (Chen et al., 2020).
  • Applied to image–caption pairs, the same loss produced CLIP, whose encoders support zero-shot image classification and are used to condition text-to-image models (Radford et al., 2021; Ramesh et al., 2022).
  • The same framework is used for sentence embeddings, audio–text models and the encoders of multimodal models.

Questions to Think About

  • Why do more negatives, for example from larger batches, tighten the mutual-information bound and often give better representations?
  • What happens when two examples of the same class are treated as negatives of each other, and how much does correcting for these false negatives help (Chuang et al., 2020)?

Core Math

Key Equation
L=−log⁡exp⁡(sim(zi,zj+)/τ)∑kexp⁡(sim(zi,zk)/τ)\mathcal{L} = -\log \frac{\exp(\text{sim}(z_i, z_j^+) / \tau)}{\sum_k \exp(\text{sim}(z_i, z_k) / \tau)}

Contrastive learning trains an encoder so that two views of the same example (a positive pair) get similar embeddings, while views of different examples (negatives) do not.

InfoNCE loss for an anchor ziz_i with positive zj+z_j^+:

L=−log⁡exp⁡(sim(zi,zj+)/τ)∑k=1Nexp⁡(sim(zi,zk)/τ)\mathcal{L} = -\log \frac{\exp(\text{sim}(z_i, z_j^+) / \tau)}{\sum_{k=1}^{N} \exp(\text{sim}(z_i, z_k) / \tau)}

where the sum runs over the positive and N−1N-1 negatives, sim\text{sim} is usually cosine similarity and τ\tau is a temperature. In SimCLR the two views are random augmentations of the same image, and the negatives are the other images in the batch (Chen et al., 2020).

The loss is a cross-entropy for picking the positive out of NN candidates. If the positive pair is sampled from the joint distribution and the N−1N-1 negatives are independently drawn from the opposite view's marginal, van den Oord et al. (2018) show I≥log⁡N−E[L]I \ge \log N - \mathbb{E}[\mathcal L]. The expectation is over those draws, not one observed batch; increasing NN raises the bound's possible ceiling.

CLIP applies the same loss to image–text pairs, in both directions (Radford et al., 2021):

LCLIP=12(Li2t+Lt2i)\mathcal{L}_{CLIP} = \frac{1}{2}\left( \mathcal{L}_{i2t} + \mathcal{L}_{t2i} \right)

Key Papers

Representation Learning with Contrastive Predictive Coding

van den Oord, Li & Vinyals2018arXiv
Read paper →

A Simple Framework for Contrastive Learning of Visual Representations

Chen et al.2020ICML
Read paper →

Learning Transferable Visual Models From Natural Language Supervision

Radford et al.2021ICML
Read paper →

Debiased Contrastive Learning

Chuang et al.2020NeurIPS
Read paper →

Connections

Prerequisites

EmbeddingsMaximum Likelihood