Self-supervised learning obtains training targets from the data rather than separate annotations. BERT uses masked language modeling, while GPT-style models such as GPT-3 and LLaMA use next-token prediction (Devlin et al., 2019; Brown et al., 2020; Touvron et al., 2023).
Concept 65 of 100Self-Supervised LearningRepresentations · Study phase 5: Representation & interpretability
Question
Why does predicting a masked or next word require knowledge of grammar, facts and context, so that the learned representations help with tasks the model was never trained on?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Self-supervised learning obtains training targets from the data rather than separate annotations. BERT uses masked language modeling, while GPT-style models such as GPT-3 and LLaMA use next-token prediction (Devlin et al., 2019; Brown et al., 2020; Touvron et al., 2023).
Because the targets come from the data, pretraining can use far more text or images than any labeled dataset contains; labeled data is then needed only for fine-tuning and evaluation.
In vision, MAE pretraining let a ViT-Huge model reach 87.8% top-1 accuracy on ImageNet-1K using only ImageNet-1K data, and its features transferred to other tasks better than those from supervised pretraining (He et al., 2022).
Questions to Think About
Why does MAE hide 75% of image patches while BERT selects only 15% of tokens for prediction, and what does that say about how much redundancy images and text contain (He et al., 2022)?
How do contrastive and other joint-embedding methods avoid the trivial solution in which every input is mapped to the same embedding (Balestriero et al., 2023)?
Core Math
Key Equation
L=−E[logp(xmasked∣xvisible)]
Self-supervised learning makes training targets from the data itself, typically by hiding part of each input and training the model to predict it.
Masked language modelling (BERT) selects a random subset M containing 15% of the tokens for prediction. The corrupted sequence x~ replaces each selected token with [MASK] 80% of the time, with a random token 10% of the time, and leaves it unchanged 10% of the time (Devlin et al., 2019):
LMLM=−Ex,M,x~[i∈M∑logp(xi∣x~)]
Next-token prediction (GPT-style language models) predicts each token from the ones before it:
LNTP=−t=1∑Tlogp(xt∣x<t)
Masked image modelling with masked autoencoders (MAE) hides a large fraction of the image patches, typically 75%, and reconstructs their pixels (He et al., 2022).
Contrastive methods (SimCLR, CLIP) instead train embeddings so that two views of the same example are similar and views of different examples are not:
L=−log∑kexp(sim(zi,zk)/τ)exp(sim(zi,zi+)/τ)
Key Papers
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding