Continuous Function

Foundation reference

Multimodal Foundations: Vision Encoders, Contrastive Learning & Cross-Attention Fusion

Contrastively trained image encoders such as CLIP's are a common front end for multimodal language models, and CLIP embeddings condition text-to-image models such as DALL·E 2 (Liu et al., 2023; Ramesh et al., 2022).

Concept 32 of 100RepresentationsStudy phase 8
Open the demoAll concepts
Key equation
Li=12(−log⁡esii∑jesij−log⁡esii∑jesji)\mathcal{L}_i = \frac{1}{2}\left(-\log\frac{e^{s_{ii}}}{\sum_j e^{s_{ij}}} - \log\frac{e^{s_{ii}}}{\sum_j e^{s_{ji}}}\right)
Reading map and next steps

Study guide

Where to start

Concept 32 of 100Multimodal PretrainingRepresentations · Study phase 8: Scaling, theory & multimodal
Question

Why does contrastive training on whole images and captions give good global semantics but weak localization, and what extra losses or architectures fix this?

Open the demoGo to the mathAll conceptsNext concept
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.

Before you move a control in the demo, say what you expect to change.

CheckCheck against the math and the papers.

Compare what the demo shows with the key equation and with the papers cited below.

GeneralizeName what stays the same.

Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.

ContinueFollow the connections.

Use the prerequisites and related concepts at the end of the page to fill the next gap.

Why It Matters for Modern Models

  • Contrastively trained image encoders such as CLIP's are a common front end for multimodal language models, and CLIP embeddings condition text-to-image models such as DALL·E 2 (Liu et al., 2023; Ramesh et al., 2022).
  • SigLIP 2 extends image–text training with captioning-based pretraining, self-supervised losses (self-distillation and masked prediction) and online data curation, to produce better encoders for vision–language models (Tschannen et al., 2025).
  • CLOC adds region-level objectives because image-level contrastive training alone does not give the localized features needed for grounding (Chen et al., 2024).
  • The shared embedding space links this concept to diffusion and to representation learning: it is how text can steer an image generator.
  • Zero-shot classification follows directly: embed each class name as a caption and choose the one closest to the image (Radford et al., 2021).

Questions to Think About

  • Why do batch size and the number of negatives matter so much for contrastive training, and how does SigLIP's sigmoid loss change that?
  • What does a dual encoder gain in retrieval speed, and what does it lose in grounding, compared with fusing image and text tokens inside one model?
  • How does the temperature τ change the loss: why does a low τ concentrate the gradient on the hardest negatives while a high τ spreads it out?
  • Images and text do not start out aligned; what in the loss pulls matched pairs together and pushes mismatched pairs apart?

Interactive Demo

Core Math

If you prefer intuition first, try the demo above, then come back for the math.

Key Equation
Li=12(−log⁡esii∑jesij−log⁡esii∑jesji)\mathcal{L}_i = \frac{1}{2}\left(-\log\frac{e^{s_{ii}}}{\sum_j e^{s_{ij}}} - \log\frac{e^{s_{ii}}}{\sum_j e^{s_{ji}}}\right)

Vision–language pretraining trains an image encoder and a text encoder so that matching images and captions land close together in a shared embedding space. The encoders are then used to feed images into multimodal language models and to condition image generators on text.

Dual-encoder similarity (normalized embeddings):

ui=fI(Ii)∥fI(Ii)∥,vj=fT(Tj)∥fT(Tj)∥,sij=ui⊤vjτu_i = \frac{f_I(I_i)}{\|f_I(I_i)\|}, \quad v_j = \frac{f_T(T_j)}{\|f_T(T_j)\|}, \quad s_{ij} = \frac{u_i^\top v_j}{\tau}

Here sijs_{ij} is the cosine similarity between image ii and caption jj, divided by a temperature τ\tau.

CLIP's contrastive loss (symmetric cross-entropy over a batch):

Li=12(−log⁡esii∑jesij−log⁡esii∑jesji)\mathcal{L}_i = \frac{1}{2}\left( -\log\frac{e^{s_{ii}}}{\sum_j e^{s_{ij}}} - \log\frac{e^{s_{ii}}}{\sum_j e^{s_{ji}}} \right)

averaged over the images ii in the batch. Each image must pick out its own caption among all captions in the batch, and each caption its own image; the other pairs act as negatives (Radford et al., 2021). SigLIP replaces the softmax with an independent sigmoid loss on every image–caption pair, which removes the normalization over the whole batch (Zhai et al., 2023).

Feeding images to a language model:

Flamingo lets text tokens attend to image features through added cross-attention layers (Alayrac et al., 2022):

CrossAttn(HT,HI)=softmax ⁣((HTWQ)(HIWK)⊤dk)(HIWV)\text{CrossAttn}(H_T, H_I) = \text{softmax}\!\left(\frac{(H_TW_Q)(H_IW_K)^\top}{\sqrt{d_k}}\right)(H_IW_V)

Many later models instead project the image-patch embeddings into the language model's input sequence and let ordinary self-attention mix them with the text (Liu et al., 2023).

Key Papers

Learning Transferable Visual Models From Natural Language Supervision

Radford et al.2021ICML
Read paper →

Sigmoid Loss for Language Image Pre-Training

Zhai et al.2023ICCV
Read paper →

Flamingo: a Visual Language Model for Few-Shot Learning

Alayrac et al.2022NeurIPS
Read paper →

Visual Instruction Tuning

Liu et al.2023NeurIPS
Read paper →

SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

Tschannen et al.2025arXiv
Read paper →

Contrastive Localized Language-Image Pre-Training

Chen et al.2024arXiv
Read paper →

Connections

Prerequisites

AttentionEmbeddingsMaximum LikelihoodDiffusion

Leads to

Lie Groups

Where to go next

Lie Groups builds directly on Multimodal Pretraining.

Lie Groups
Other connections (7)

Where does it break down?

  • Long patch sequences: where Multimodal Pretraining breaks down, taken up in Long Context. Open Long Context

    Images and videos become long sequences of patch or frame tokens, often far longer than the text that accompanies them, so they use up the context window and the key-value cache much faster than text alone.

Where else does the same idea appear?

  • Text-conditional generation: the same technique appears in Diffusion. Open Diffusion

    Text-to-image diffusion models such as Stable Diffusion condition the denoiser on text embeddings through cross-attention (Rombach et al., 2022), so the text guides every denoising step.

  • Shared attention: the same technique appears in Attention. Open Attention

    Multimodal models reuse the attention mechanism across modalities. Some interleave image and text tokens in one sequence and use ordinary self-attention; others add cross-attention layers so that one modality attends to the other (Alayrac et al., 2022).

  • Everything as a sequence: an analogy between Multimodal Pretraining and Tokenization. Open Tokenization

    Many multimodal models turn every input into a sequence of vectors that a transformer can process: image patches, pieces of audio, and text tokens. The sequence model is shared; only the way each input is cut into pieces differs.

  • Cross-attention fusion: the same technique appears in Attention. Open Attention

    Some multimodal models fuse modalities with cross-attention: in Flamingo, text tokens attend to image features through added cross-attention layers (Alayrac et al., 2022). Others use none: CLIP encodes images and text separately and only compares the two embeddings.

  • Patch = token: an analogy between Multimodal Pretraining and Tokenization. Open Tokenization

    Vision transformers cut an image into patches and treat each patch embedding like a token embedding, so the same transformer can process images and text (Dosovitskiy et al., 2021). Unlike text tokens, patches are not drawn from a fixed vocabulary.

  • InfoNCE similarity matrix: the same technique appears in Embeddings. Open Embeddings

    CLIP training is contrastive representation learning across two modalities: build the matrix of image–text similarities in a batch and apply a softmax cross-entropy along each row and each column (Radford et al., 2021). SigLIP keeps the similarity matrix but replaces the softmax with an independent sigmoid loss on every pair (Zhai et al., 2023).