Machine Learning

Self-Supervised Vision: SimCLR, MoCo, DINO, and MAE

Self-supervised vision trains encoders from unlabeled images by making views agree, comparing positives against negatives, matching teacher targets, or reconstructing masked patches.

status: reviewimportance: importantdifficulty 4/5math: graduateread: 20mlive demo

Back to Machine Learning Next: Vision-Language Models: CLIP and BLIP-Style Systems (review)

Concept Structure

Self-Supervised Vision: SimCLR, MoCo, DINO, and MAE

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

3prerequisites

2next concepts

5related links

Learner Contract

What this page should let you do.

You are here becauseSelf-supervised vision trains encoders from unlabeled images by making views agree, comparing positives against negatives, matching teacher targets, or reconstructing masked patches.

This Machine Learning concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

Before thisAutoencoders and Denoising Autoencoders

3 prerequisites listed; refresh them before leaning on the math or code.

Autoencoders and Denoising Autoencoders Vision Transformers (review)Representation Learning & Embedding Geometry

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextVision-Language Models: CLIP and BLIP-Style Systems (review)

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Vision-Language Models: CLIP and BLIP-Style Systems (review)

Claim/source review status

Substantive review recorded

2/2 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.

Claims2/2 reviewed

Sources5 cited

Codeattached

Demolive

Reviewed2026-07-04

Updatedpage 2026-07-04

Learning item flow

4/4 sections readyAsk about this Research room

ConceptSelf-Supervised Vision: SimCLR, MoCo, DINO, and MAEMachine Learning EquationSelf-Supervised Vision: SimCLR, MoCo, DINO, and MAE equation 1Exact equation CodeSelf-Supervised Vision: SimCLR, MoCo, DINO, and MAE runnable code 1Exact runnable code DemoSelf-Supervised Vision: SimCLR, MoCo, DINO, and MAE interactive demoInteractive demo ClaimSelf-supervised vision replaces manual class labels with pretext sign...Exact claim check SourceA Simple Framework for Contrastive Learning of Visual RepresentationsExact source

ConceptSelf-Supervised Vision: SimCLR, MoCo, DINO, and MAEMachine Learning

5 sources attachedLocal snapshot ready

concept:machine-learning/self-supervised-vision-simclr-moco-dino-mae

Codenearby Predictbefore reveal Roomitem handoff

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

Supervised image classification says: here is an image, here is the class label, update the model so the image points toward that label.

Self-supervised vision asks a quieter question:

what useful prediction problem can the image create for itself?

That is the shared thread across SimCLR, MoCo, DINO, and MAE. They are not four random names to memorize. They are four ways to replace a human label with a training signal:

SimCLR: make two augmented views of the same image agree while separating them from other images in the batch.
MoCo: keep the contrastive idea, but maintain a queue of encoded negatives and a slowly updated key encoder so the dictionary is large and consistent.
DINO: train a student network to match a teacher network's output distribution across different crops, with the teacher updated by momentum rather than labels.
MAE: hide many image patches, encode the visible patches, and train a decoder to reconstruct the missing pixels or patch targets.

The practical reason this matters is representation transfer. After pretraining, the encoder is not used because it solved "same image crop" or "fill these pixels" as an end in itself. It is used because those tasks can pressure the encoder to preserve visual structure that later helps classification, retrieval, detection, segmentation, or other downstream tasks.

The caveat is just as important. A pretext task is a pressure, not a proof of understanding. The learned representation can inherit shortcut augmentations, dataset bias, texture dependence, or poor transfer to a new domain. The right mental model is:

self-supervision manufactures a learning signal; downstream evaluation checks whether the representation became useful.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1 $v_i = t_i(x), \qquad v_j = t_j(x).$ Equation 2 $z_i = \frac{g(f(v_i))}{\lVert g(f(v_i)) \rVert_2}, \qquad z_j = \frac{g(f(v_j))}{\lVert g(f(v_j)) \rVert_2}.$

Let an image $x$ produce two augmented views:

v_i = t_i(x), \qquad v_j = t_j(x).

An encoder and projection head map those views to normalized vectors:

z_i = \frac{g(f(v_i))}{\lVert g(f(v_i)) \rVert_2}, \qquad z_j = \frac{g(f(v_j))}{\lVert g(f(v_j)) \rVert_2}.

In a SimCLR-style contrastive objective, $(i,j)$ is a positive pair. Other views in the batch become negatives. A common normalized temperature-scaled loss for anchor $i$ is:

\mathcal L_i = -\log \frac{\exp(\operatorname{sim}(z_i,z_j)/\tau)} {\sum_{k\ne i}\exp(\operatorname{sim}(z_i,z_k)/\tau)}.

The numerator pulls the positive pair together. The denominator makes nearby negatives expensive. The temperature $\tau$ controls how sharp that competition is.

MoCo keeps the same contrastive shape but changes where negatives come from. Instead of depending only on the current batch, it keeps a queue of previous key vectors. To keep those keys compatible, the key encoder is updated by a moving average:

\theta_k \leftarrow m\theta_k + (1-m)\theta_q.

DINO removes explicit negatives from the central objective. A student distribution $p_s$ is trained to match a teacher distribution $p_t$ across crops:

\mathcal L_{\mathrm{DINO}} = - \sum_c p_t(c)\log p_s(c).

The teacher is also updated by momentum. Centering, sharpening, multi-crop training, and implementation details matter because a naive student-teacher match can collapse to uninformative constant outputs.

MAE uses a different signal. Split an image into patches, mask a random set $M$ , and keep visible patches $V$ :

V \cup M = \{1,\ldots,N\}, \qquad V \cap M = \varnothing.

The encoder processes visible patches. The decoder reconstructs masked targets:

\mathcal L_{\mathrm{MAE}} = \frac{1}{|M|} \sum_{p\in M} \ell(\widehat x_p, x_p).

For a $4\times4$ teaching grid with $N=16$ and a $75\%$ mask ratio:

|M|=12,\qquad |V|=4.

The encoder sees only one quarter of the patches. The decoder has to use context from visible patches plus mask tokens to reconstruct the hidden targets.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Runnable code 1import numpy as np def unit(v): return v / (np.linalg.norm(v) + 1e-9) def info_nce(anchor, po...python

import numpy as np

def unit(v):
    return v / (np.linalg.norm(v) + 1e-9)

def info_nce(anchor, positive, negatives, tau=0.2):
    candidates = [positive] + negatives
    sims = np.array([unit(anchor) @ unit(v) for v in candidates])
    logits = sims / tau
    probs = np.exp(logits - logits.max())
    probs = probs / probs.sum()
    loss = -np.log(probs[0])
    return sims, probs[0], loss

anchor = np.array([0.92, 0.28, 0.10])
positive = np.array([0.86, 0.33, 0.12])
negatives = [
    np.array([-0.20, 0.90, 0.10]),
    np.array([0.10, -0.80, 0.50]),
    np.array([-0.70, -0.20, 0.30]),
    np.array([0.25, 0.15, -0.95]),
]

sims, positive_probability, loss = info_nce(anchor, positive, negatives)
print("cosine similarities:", np.round(sims, 3))
print("positive probability:", round(float(positive_probability), 3))
print("loss:", round(float(loss), 3))

patches = 16
mask_ratio = 0.75
masked = int(patches * mask_ratio)
visible = patches - masked
print({"visible_patches": visible, "masked_targets": masked})

This small witness is deliberately narrow. It does not train an image model. It exposes two mechanics that otherwise become hand-wavy:

the contrastive loss gives the positive view a probability among candidates;
a high MAE mask ratio means the encoder sees a small visible subset while the decoder predicts the hidden patch targets.

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the Self-Supervised Vision Objective Lab to predict the hidden training signal before seeing the proof:

Positive pair: decide what should happen to two augmented views from the same unlabeled image.
Negative bank: decide why a contrastive queue changes the candidate set.
Masked reconstruction: decide what MAE asks the model to predict when most patches are hidden.

Before reveal, the positive probability, negative-bank count, teacher target, and mask ratio stay locked. After reveal, compare the route you chose with the numeric witness. The point is to connect objective mechanics, not to memorize a leaderboard.

Live Concept Demo

Explore Self-Supervised Vision: SimCLR, MoCo, DINO, and MAE

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Self-Supervised Vision: SimCLR, MoCo, DINO, and MAE should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Self-supervised vision trains encoders from unlabeled images by making views agree, comparing positives against negatives, matching teacher targets, or reconstructing masked patches.

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Self-Supervised Vision: SimCLR, MoCo, DINO, and MAE should make visible.

Visual Inquiry

Make the image answer a mathematical question

Self-supervised vision trains encoders from unlabeled images by making views agree, comparing positives against negatives, matching teacher targets, or reconstructing masked patches.

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Self-Supervised Vision: SimCLR, MoCo, DINO, and MAE easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

paper · 2020A Simple Framework for Contrastive Learning of Visual RepresentationsChen, Kornblith, Norouzi, and Hinton

Primary support for SimCLR's augmented-view positive pairs, contrastive loss, projection head, and the importance of augmentations, batch size, and training length.

Open source

paper · 2020Momentum Contrast for Unsupervised Visual Representation LearningHe, Fan, Wu, Xie, and Girshick

Primary support for the dictionary lookup view, queue of negatives, and moving-average key encoder.

Open source

paper · 2021Emerging Properties in Self-Supervised Vision TransformersCaron et al.

Primary support for DINO as self-distillation without labels using a momentum teacher and multi-crop training with ViTs.

Open source

paper · 2021Masked Autoencoders Are Scalable Vision LearnersHe, Chen, Xie, Li, Dollar, and Girshick

Primary support for masking random image patches, encoding visible patches only, reconstructing missing pixels, asymmetric encoder-decoder design, and high mask ratios such as 75%.

Open source

paper · 2018Representation Learning with Contrastive Predictive Codingvan den Oord, Li, and Vinyals

Reference for contrastive representation learning and InfoNCE-style losses that score positives against negative samples.

Open source

Claim Review

Self-supervised vision trains encoders from unlabeled images by making views agree, comparing positives against negatives, matching teacher targets, or reconstructing masked patches.

Status2 substantive reviews recorded

Claims without a substantive review badge still need exact source-support review.

Sources5 references

chen-2020-simclr, he-2020-moco, caron-2021-dino, he-2021-mae, oord-2018-cpc

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedSelf-supervised vision replaces manual class labels with pretext signals such as augmented-view agreement, contrastive dictionary lookup, teacher-student distribution matching, or masked patch reconstruction.Claim metadata: source checked

The references support augmented positive views, contrastive negatives, a dynamic queue with momentum keys, DINO-style self-distillation without labels, and MAE-style masked patch reconstruction.

Sources: A Simple Framework for Contrastive Learning of Visual Representations, Momentum Contrast for Unsupervised Visual Representation Learning, Emerging Properties in Self-Supervised Vision Transformers, Masked Autoencoders Are Scalable Vision Learners, Representation Learning with Contrastive Predictive CodingThis page teaches objective mechanics, not a benchmark ranking, universal superiority claim, or exhaustive survey of all self-supervised vision methods.A bounded review summary is present; still check caveats and exact reference scope.

Checked SimCLR, MoCo, DINO, MAE, and CPC for the bounded claim that these methods define training signals from unlabeled image views, queues or teachers, and masked patches. GPT Pro publication critique remains pending because the Oracle wrapper and remote Chrome port are unavailable on this workstation.

Reviewer: codex-local-primary-reference-audit; reviewed 2026-07-04

source-span-chen-2020-simclr source-span-he-2020-moco source-span-caron-2021-dino source-span-he-2021-mae Equation 1 Equation 2 Runnable code 1 Interactive demo

Substantively reviewedA self-supervised vision encoder is usually evaluated by transferring the learned representation to downstream tasks, but the pretext objective itself does not guarantee semantic understanding or robustness.Claim metadata: source checked

The papers evaluate representations with linear probes, transfer, or downstream tasks; they do not make the pretext objective itself equivalent to complete scene understanding.

Sources: A Simple Framework for Contrastive Learning of Visual Representations, Momentum Contrast for Unsupervised Visual Representation Learning, Emerging Properties in Self-Supervised Vision Transformers, Masked Autoencoders Are Scalable Vision LearnersTransfer quality depends on data, augmentation policy, architecture, compute, objective details, evaluation protocol, and downstream task shift.A bounded review summary is present; still check caveats and exact reference scope.

Checked the method papers for representation transfer framing and downstream evaluation context, then bounded the learner-facing caveat to avoid overstating the pretext task.

Reviewer: codex-local-primary-reference-audit; reviewed 2026-07-04

source-span-chen-2020-simclr source-span-he-2020-moco source-span-caron-2021-dino source-span-he-2021-mae Interactive demo

Source support candidates

paper 2020A Simple Framework for Contrastive Learning of Visual Representations

Primary support for SimCLR's augmented-view positive pairs, contrastive loss, projection head, and the importance of augmentations, batch size, and training length.

paper 2020Momentum Contrast for Unsupervised Visual Representation Learning

Primary support for the dictionary lookup view, queue of negatives, and moving-average key encoder.

paper 2021Emerging Properties in Self-Supervised Vision Transformers

Primary support for DINO as self-distillation without labels using a momentum teacher and multi-crop training with ViTs.

paper 2021Masked Autoencoders Are Scalable Vision Learners

Primary support for masking random image patches, encoding visible patches only, reconstructing missing pixels, asymmetric encoder-decoder design, and high mask ratios such as 75%.

Mechanism checks

Equation 1

v_i = t_i(x), \qquad v_j = t_j(x).

Equation 2

z_i = \frac{g(f(v_i))}{\lVert g(f(v_i)) \rVert_2}, \qquad z_j = \frac{g(f(v_j))}{\lVert g(f(v_j)) \rVert_2}.

Runnable code 1import numpy as np def unit(v): return v / (np.linalg.norm(v) + 1e-9) def info_nce(anchor, po...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Self-supervised vision trains encoders from unlabeled images by making views agree, comparing positives against negatives, matching teacher targets, or reconstructing masked patches.

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in Self-Supervised Vision: SimCLR, MoCo, DINO, and MAE.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Grounded research drawerClose

ConceptSelf-Supervised Vision: SimCLR, MoCo, DINO, and MAEMachine Learning

Runnable code comparisonSelf-Supervised Vision: SimCLR, MoCo, DINO, and MAE runnable code 1candidates = [positive] + negatives Prediction before revealSelf-Supervised Vision: SimCLR, MoCo, DINO, and MAE interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes Self-Supervised Vision: SimCLR, MoCo, DINO, and MAE click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptMachine Learning

Self-Supervised Vision: SimCLR, MoCo, DINO, and MAE

Attached question

What is the smallest example that makes Self-Supervised Vision: SimCLR, MoCo, DINO, and MAE click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays in this browser, attached to the selected learning item.

Draft noteNext action

No local draft saved.

Evidence to inspect

References to inspect: attached references on this page.
Definition, prerequisite, and contrast concept links
The equation or runnable code that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Self-Supervised Vision: SimCLR, MoCo, DINO, and MAE Selected item key: recorded for copy. Context: Machine Learning Page anchor: recorded for copy. Open question: What is the smallest example that makes Self-Supervised Vision: SimCLR, MoCo, DINO, and MAE click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context

concept/concept-notebook/machine-learning/self-supervised-vision-simclr-moco-dino-mae
concept:machine-learning/self-supervised-vision-simclr-moco-dino-mae

Learning Map

Before / Now / Try / Next

BeforeAutoencoders and Denoising Autoencoders

NowIntuition -> Math -> Code -> Demo

TryManipulate one control and predict the visible change.

NextVision-Language Models: CLIP and BLIP-Style Systems (review)

Intuitionready
Mathready
Codeready
Interactive Demoready

AI Companion

Ask beside the selected object

Self-supervised vision trains encoders from unlabeled images by making views agree, comparing positives against negatives, matching teacher targets, or reconstructing masked patches.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Machine Learning concept. Learning surface: Self-Supervised Vision: SimCLR, MoCo, DINO, and MAE. What this page says: Self-supervised vision trains encoders from unlabeled images by making views agree, comparing positives against negatives, matching teacher targets, or reconstructing masked patches. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

Machine Learning

machine-learningcomputer-visionself-supervised-learningcontrastive-learningmasked-autoencodervision-transformer

Prerequisites

Autoencoders and Denoising Autoencoders Vision Transformers (review)Representation Learning & Embedding Geometry

Leads To

Vision-Language Models: CLIP and BLIP-Style Systems (review)Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps (review)

Image Classification Pipeline (review)Data Augmentation and Transfer Learning (review)Object Detection: R-CNN, YOLO, SSD, and DETR (review)CNN Architecture Families (review)Modern CNN Block Mechanics (review)

Within this domain