Continuous Function

Machine Learning

Vision Transformers

Vision Transformers turn an image into patch tokens, add positional information, and use self-attention plus a class representation for image classification.

status: reviewimportance: importantdifficulty 4/5math: graduateread: 18mlive demo
Back to Machine LearningNext: Vision-Language Models: CLIP and BLIP-Style Systems (review)

Concept Structure

Vision Transformers

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

4prerequisites
3next concepts
5related links

Learner Contract

What this page should let you do.

You are here becauseVision Transformers turn an image into patch tokens, add positional information, and use self-attention plus a class representation for image classification.

This Machine Learning concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Test the linkManipulate one control and predict the visible change.Then continue to Vision-Language Models: CLIP and BLIP-Style Systems (review)

Claim/source review status

Substantive review recorded

2/2 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.
Claims2/2 reviewed
Sources4 cited
Codeattached
Demolive
Reviewed2026-07-04
Updatedpage 2026-07-04

Learning item flow

4/4 sections readyAsk about thisResearch room
ConceptVision TransformersMachine LearningEquationVision Transformers equation 1Exact equationCodeVision Transformers runnable code 1Exact runnable codeDemoVision Transformers interactive demoInteractive demoClaimA Vision Transformer turns an image into a sequence by splitting it i...Exact claim checkSourceAn Image is Worth 16x16 Words: Transformers for Image Recognition at...Exact source
ConceptVision TransformersMachine Learning
4 sources attachedLocal snapshot ready
concept:machine-learning/vision-transformers
CodenearbyPredictbefore revealRoomitem handoff
01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.
01

01

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

A CNN starts with a strong promise: nearby pixels should be mixed first, the same filter should scan across the image, and translation should matter less than the local pattern. A Vision Transformer makes a different opening move. It cuts the image into patches, treats those patches like a sequence of tokens, adds information about where each token came from, and lets self-attention compare patches globally.

That swap is powerful, but easy to misread. A ViT is not "attention sprinkled on an image." It is a specific tokenization contract:

  • split an image into non-overlapping patches,
  • flatten each patch,
  • linearly project patches into a common embedding width,
  • prepend a learned class token or use an equivalent pooling route,
  • add positional embeddings,
  • run Transformer encoder layers,
  • send the class representation to a classifier head.

The bridge from CNNs is the question of where structure enters the model. CNNs build locality and translation sharing directly into convolution. A plain ViT asks the data, positional signal, and attention layers to learn more of that structure. That is why patch size, token count, positional information, and data regime are not side details.

02

02

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt
Equation 1X∈RH×W×3.X\in\mathbb R^{H\times W\times 3}.Equation 2N=HP⋅WP.N=\frac{H}{P}\cdot\frac{W}{P}.

Let an RGB image have shape

X∈RH×W×3.X\in\mathbb R^{H\times W\times 3}.

Choose a square patch size PP that divides HH and WW. The number of image patches is

N=HP⋅WP.N=\frac{H}{P}\cdot\frac{W}{P}.

For the common teaching example H=W=224H=W=224 and P=16P=16:

N=14⋅14=196.N=14\cdot14=196.

Each patch is flattened into a vector of length

P2⋅3.P^2\cdot 3.

For P=16P=16, that patch vector length is

162⋅3=768.16^2\cdot3=768.

The patch projection maps each flattened patch to an embedding vector in RD\mathbb R^D:

zi=xiE,E∈R(P2⋅3)×D.z_i = x_iE,\qquad E\in\mathbb R^{(P^2\cdot3)\times D}.

Then the model prepends a learned class token zclsz_{\text{cls}} and adds positional embeddings:

Z0=[zcls;z1;…;zN]+Epos.Z_0 = [z_{\text{cls}}; z_1; \ldots; z_N] + E_{\text{pos}}.

So the sequence length entering the encoder is

L=N+1.L=N+1.

For 224×224224\times224 images with P=16P=16, this gives

L=197.L=197.

If D=768D=768, the encoder input has shape

197×768.197\times768.

The self-attention budget grows with token pairs. A useful first-order count is

L2.L^2.

That means changing patch size is not cosmetic:

P=32:L=50,L2=2500,P=32:\quad L=50,\quad L^2=2500,

while

P=8:L=785,L2=616225.P=8:\quad L=785,\quad L^2=616225.

Smaller patches preserve more spatial detail but create many more token pairs for attention to compare. Larger patches reduce the sequence length but force each token to summarize a larger region before attention starts.

03

03

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt
Runnable code 1def vit_shapes(image_size=224, patch_size=16, channels=3, embed_dim=768): patches_per_side =...python
def vit_shapes(image_size=224, patch_size=16, channels=3, embed_dim=768):
    patches_per_side = image_size // patch_size
    patch_count = patches_per_side ** 2
    token_count = patch_count + 1  # class token
    patch_vector = patch_size * patch_size * channels
    projection_params = patch_vector * embed_dim
    attention_pairs = token_count * token_count
    return {
        "patches": patch_count,
        "tokens": token_count,
        "embedding_shape": (token_count, embed_dim),
        "patch_vector": patch_vector,
        "projection_params": projection_params,
        "attention_pairs": attention_pairs,
    }

for p in [32, 16, 8]:
    print(p, vit_shapes(patch_size=p))

This is a shape and token-budget witness. It does not train a ViT. Its job is to make the first contract visible: patch size controls how many tokens enter attention; the class token adds one more token; the embedding width controls the vector dimension; and attention compares token pairs.

04

04

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the Patch Token Lab as a reveal-gated shape check:

  • Token count: predict whether the encoder sees patches only, class plus patches, or embedding width as sequence length.
  • Position role: predict what positional embeddings add after patch projection.
  • Class token: predict why the class token is prepended before the encoder.

Before reveal, the token count, embedding shape, and caveat cards stay locked. After reveal, compare the patch arithmetic, the encoder input shape, and the attention-pair count for patch sizes 88, 1616, and 3232. The point is not to memorize one ViT configuration; it is to see which quantities change when an image becomes a token sequence.

Live Concept Demo

Explore Vision Transformers

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned
Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Vision Transformers should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Vision Transformers turn an image into patch tokens, add positional information, and use self-attention plus a class representation for image classification.

Prediction open01 / Intuition
Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Vision Transformers should make visible.

Visual Inquiry

Make the image answer a mathematical question

Vision Transformers turn an image into patch tokens, add positional information, and use self-attention plus a class representation for image classification.

4/4 stages readyLive demo connected
Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Vision Transformers easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

paper · 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleDosovitskiy et al.

Primary source for splitting images into fixed-size patches, linearly embedding them, prepending a classification token, adding positional embeddings, and applying a Transformer encoder to image patches.

Open source
book · 2026Dive into Deep Learning: Vision TransformerZhang, Lipton, Li, and Smola

Teaching reference for patch embeddings, class token, positional embeddings, transformer encoder blocks, and the classification head.

Open source
course-notes · 2024CS231n: Attention and TransformersStanford CS231n

Course reference that situates Vision Transformers in deep computer vision and compares ViT-style models with ResNet-era convolutional models.

Open source
paper · 2017Attention Is All You NeedVaswani et al.

Source for Transformer encoder ingredients used by ViT: multi-head self-attention, feed-forward blocks, residual pathways, layer normalization context, and positional information.

Open source

Claim Review

Vision Transformers turn an image into patch tokens, add positional information, and use self-attention plus a class representation for image classification.

Status2 substantive reviews recorded

Claims without a substantive review badge still need exact source-support review.

Sources4 references

dosovitskiy-2020-vit, d2l-vision-transformer, cs231n-2024-attention-transformers, vaswani-2017-attention

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedA Vision Transformer turns an image into a sequence by splitting it into fixed-size patches, projecting each patch to an embedding vector, prepending a learned class token, adding positional embeddings, and running a Transformer encoder over the resulting tokens.Claim metadata: source checked

The references support the patch-to-token pipeline, class-token classification route, positional embeddings, and use of Transformer encoder layers over image patch tokens.

Sources: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, Dive into Deep Learning: Vision Transformer, CS231n: Attention and Transformers, Attention Is All You NeedThis page covers the basic ViT classifier mechanism, not hierarchical/windowed variants, detection or segmentation transformers, masked autoencoder training, CLIP-style contrastive learning, or a benchmark ranking.A bounded review summary is present; still check caveats and exact reference scope.

Checked the ViT paper, D2L ViT chapter, CS231n transformer slides, and the original Transformer paper for patchification, linear patch embeddings, class token, positional embeddings, and encoder-style self-attention. GPT Pro publication critique remains pending because the Oracle wrapper and remote Chrome port are unavailable on this workstation.

Reviewer: codex-local-primary-reference-audit; reviewed 2026-07-04
source-span-dosovitskiy-2020-vitsource-span-d2l-vision-transformerEquation 1Equation 2Runnable code 1Interactive demo
Substantively reviewedPatch size controls the number of image tokens and therefore the self-attention budget; compared with CNNs, a plain ViT has weaker built-in locality and translation bias, so the data and regularization setting matter.Claim metadata: source checked

The references support the patch-count formula, the extra class token, the quadratic token-pair pressure of self-attention, and the caution that plain ViTs trade away some CNN locality/translation assumptions.

Sources: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, Dive into Deep Learning: Vision Transformer, CS231n: Attention and TransformersThe token-pair count is a teaching proxy for attention budget, not a full runtime, memory, or accuracy prediction. Real performance depends on architecture variants, implementation kernels, pretraining scale, augmentation, and optimization.A bounded review summary is present; still check caveats and exact reference scope.

Checked the ViT paper, D2L ViT chapter, and CS231n transformer slides for patch-size/token-count arithmetic, self-attention over patch tokens, and the convolutional-inductive-bias caveat. GPT Pro publication critique remains pending.

Reviewer: codex-local-primary-reference-audit; reviewed 2026-07-04
source-span-dosovitskiy-2020-vitsource-span-d2l-vision-transformersource-span-cs231n-2024-attention-transformersRunnable code 1Interactive demo

Source support candidates

paper 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Primary source for splitting images into fixed-size patches, linearly embedding them, prepending a classification token, adding positional embeddings, and applying a Transformer encoder to image patches.

book 2026Dive into Deep Learning: Vision Transformer

Teaching reference for patch embeddings, class token, positional embeddings, transformer encoder blocks, and the classification head.

course-notes 2024CS231n: Attention and Transformers

Course reference that situates Vision Transformers in deep computer vision and compares ViT-style models with ResNet-era convolutional models.

paper 2017Attention Is All You Need

Source for Transformer encoder ingredients used by ViT: multi-head self-attention, feed-forward blocks, residual pathways, layer normalization context, and positional information.

Mechanism checks

Equation 1
X∈RH×W×3.X\in\mathbb R^{H\times W\times 3}.
Equation 2
N=HP⋅WP.N=\frac{H}{P}\cdot\frac{W}{P}.
Runnable code 1def vit_shapes(image_size=224, patch_size=16, channels=3, embed_dim=768): patches_per_side =...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Vision Transformers turn an image into patch tokens, add positional information, and use self-attention plus a class representation for image classification.

Readiness0/3 checks ready
Predict

Before touching the demo, predict one visible change that should happen in Vision Transformers.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Grounded research drawerClose
ConceptVision TransformersMachine Learning
Runnable code comparisonVision Transformers runnable code 1def vit_shapes(image_size=224, patch_size=16, channels=3, embed_dim=768): patches_per_side =...Prediction before revealVision Transformers interactive demoManipulate one control and predict the visible change.
Grounded room questionWhat is the smallest example that makes Vision Transformers click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.
Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptMachine Learning

Vision Transformers

Attached question

What is the smallest example that makes Vision Transformers click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action
Local action draft

This draft stays in this browser, attached to the selected learning item.

No local draft saved.
Evidence to inspect
  • References to inspect: attached references on this page.
  • Definition, prerequisite, and contrast concept links
  • The equation or runnable code that makes the concept operational
  • One demo state that shows the invariant instead of a slogan
What would resolve this
  • The learner can state the mechanism in their own words
  • The learner can name the prerequisite that would repair confusion
  • The learner can predict how the mechanism changes under one perturbation
Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Vision Transformers Selected item key: recorded for copy. Context: Machine Learning Page anchor: recorded for copy. Open question: What is the smallest example that makes Vision Transformers click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context
concept/concept-notebook/machine-learning/vision-transformers concept:machine-learning/vision-transformers