∇Continuous Function
FoundationsChecking saved investigationReading browser-local route memory before showing a continuation.

Foundation Lab

Self-Supervised Learning: Labels from Structure

Foundation of modern NLP: BERT, GPT, LLaMA all use self-supervised pretraining

Concept 65 of 100RepresentationsPhase 5
Open DemoAll Foundations
#65SSLRepresentations
key equation\mathcal{L} = -\mathbb{E}[\log p(x_{masked} | x_{visible})]

Selected Foundation Object

Keep the equation fixed; move through the evidence.

Concept 65 of 100SSLRepresentations / Phase 5: Representation & interpretability
Current question

The task (predict missing parts) forces the model to understand structure and semantics

\mathcal{L} = -\mathbb{E}[\log p(x_{masked} | x_{visible})]
PredictionCommit before the demo.

Ask what should change when the equation is manipulated, then let the visualization test that expectation.

EvidenceCompare local witness and source.

Use the runnable panel, the key equation, and canonical papers as separate forms of evidence for the same object.

InvariantName what survives notation changes.

The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.

Next moveContinue through the atlas.

Use prerequisites, dependents, and semantic links to repair the next gap without leaving the object behind.

Open demoInspect equationBrowse atlasNext foundation

Why It Matters for Modern Models

  • Foundation of modern NLP: BERT, GPT, LLaMA all use self-supervised pretraining
  • Enables learning from internet-scale unlabeled data—the key to scaling laws
  • Self-supervised vision (MAE, DINO) is closing the gap with supervised ImageNet pretraining

What Tutorials Skip

What is still poorly explained in textbooks and papers:

  • The task (predict missing parts) forces the model to understand structure and semantics
  • SSL works because predicting tokens/pixels requires modeling the full data distribution
  • Transfer learning magic: SSL features generalize because the pretext task is so hard

Interactive Visualization

Core Math (Optional Deep Dive)

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation
L=−E[log⁡p(xmasked∣xvisible)]\mathcal{L} = -\mathbb{E}[\log p(x_{masked} | x_{visible})]

Self-supervised = create supervision from data itself:

Masked Language Modeling (BERT):

LMLM=−Ex,M[∑i∈Mlog⁡p(xi∣x\M)]\mathcal{L}_{MLM} = -\mathbb{E}_{x, M}\left[\sum_{i \in M} \log p(x_i | x_{\backslash M})\right]

where MM is the set of masked positions.

Next Token Prediction (GPT):

LNTP=−∑t=1Tlog⁡p(xt∣x<t)\mathcal{L}_{NTP} = -\sum_{t=1}^T \log p(x_t | x_{<t})

Contrastive (SimCLR, CLIP):

L=−log⁡exp⁡(sim(zi,zi+)/τ)∑kexp⁡(sim(zi,zk)/τ)\mathcal{L} = -\log \frac{\exp(\text{sim}(z_i, z_i^+)/\tau)}{\sum_k \exp(\text{sim}(z_i, z_k)/\tau)}

Canonical Papers

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin et al.2019NAACL
Read paper →

Masked Autoencoders Are Scalable Vision Learners

He et al.2022CVPR
Read paper →

Connections

Prerequisites

ℒML/CE/KL◎Embeddings

Enables

📝Instruct

Next Moves

Choose the next question to carry this object forward.