Machine Learning

Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps

Segmentation turns vision into pixel-level output: semantic classes for every pixel, separate thing instances, and panoptic maps that combine stuff regions with instance IDs.

status: reviewimportance: importantdifficulty 4/5math: graduateread: 20mlive demo

Back to Machine Learning Next: Semantic, Instance, and Panoptic Segmentation (review)

Concept Structure

Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

3prerequisites

1next concepts

5related links

Learner Contract

What this page should let you do.

You are here becauseSegmentation turns vision into pixel-level output: semantic classes for every pixel, separate thing instances, and panoptic maps that combine stuff regions with instance IDs.

This Machine Learning concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

Before thisCNN Shape Mechanics: Convolution, Padding, Stride, Pooling, and Channels (review)

3 prerequisites listed; refresh them before leaning on the math or code.

CNN Shape Mechanics: Convolution, Padding, Stride, Pooling, and Channels (review)Image Classification Pipeline (review)Object Detection: R-CNN, YOLO, SSD, and DETR (review)

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextSemantic, Instance, and Panoptic Segmentation (review)

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Semantic, Instance, and Panoptic Segmentation (review)

Claim/source review status

Substantive review recorded

2/2 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.

Claims2/2 reviewed

Sources4 cited

Codeattached

Demolive

Reviewed2026-07-04

Updatedpage 2026-07-04

Learning item flow

4/4 sections readyAsk about this Research room

ConceptSegmentation: FCN, U-Net, Mask R-CNN, and Panoptic MapsMachine Learning EquationSegmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps equation 1Exact equation CodeSegmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps runnable code 1Exact runnable code DemoSegmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps interactive demoInteractive demo ClaimSemantic segmentation predicts a class for each pixel; instance segme...Exact claim check SourceFully Convolutional Networks for Semantic SegmentationExact source

ConceptSegmentation: FCN, U-Net, Mask R-CNN, and Panoptic MapsMachine Learning

4 sources attachedLocal snapshot ready

concept:machine-learning/segmentation-fcn-unet-mask-rcnn

Codenearby Predictbefore reveal Roomitem handoff

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

Image classification returns one label for a whole image. Object detection returns boxes, classes, and scores. Segmentation asks for a finer contract:

what should every pixel mean?

That single question splits into several useful output types.

Semantic segmentation: every pixel gets a class label. All cars may share the same "car" label, and all road pixels share "road."
Instance segmentation: countable objects are separated. Two cars are not only "car"; they are car #1 and car #2 with different masks.
Panoptic segmentation: every pixel is covered once. Stuff regions such as sky or road get semantic labels, while thing regions such as cars and people get both a class and an instance ID.

Architectures matter because segmentation must preserve spatial detail. A classifier can crush the image into a single vector. A segmenter has to recover where each class or object lives.

Canonical routes teach different pressures:

FCN: turn a classifier into a fully convolutional dense predictor and upsample coarse scores back to pixels.
U-Net: encode context, then decode with skip connections so high-resolution detail can re-enter the mask.
Mask R-CNN: detect object instances first, then attach a mask branch for each proposed object.

The mistake to avoid is thinking segmentation is "just coloring." The color is only the visible surface. Underneath, the model is choosing an output contract: one label per pixel, one mask per object, or a unified scene map.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1 $P \in \mathbb R^{H\times W\times C}.$ Equation 2 $\mathcal L_{\mathrm{sem}} = -\frac{1}{HW}\sum_{h=1}^{H}\sum_{w=1}^{W} \log P_{h,w,y_{h,w}}.$

Let an image have height $H$ and width $W$ . Semantic segmentation predicts a distribution over $C$ classes for each pixel:

P \in \mathbb R^{H\times W\times C}.

For pixel $(h,w)$ with class label $y_{h,w}$ , a common per-pixel cross-entropy loss is:

\mathcal L_{\mathrm{sem}} = -\frac{1}{HW}\sum_{h=1}^{H}\sum_{w=1}^{W} \log P_{h,w,y_{h,w}}.

Mask quality is often checked by overlap. For a predicted mask $\widehat M$ and target mask $M$ :

\operatorname{IoU}(\widehat M, M) = \frac{|\widehat M \cap M|} {|\widehat M \cup M|}.

A semantic mean-IoU averages this overlap across classes. Instance segmentation computes mask overlap per object candidate, usually alongside detection matching. Panoptic segmentation uses a combined view: it requires non-overlapping segments that cover the whole image, with stuff classes and thing instances both represented.

For a toy $8\times8$ scene:

road and sky are stuff classes,
car and person are thing classes,
two cars must be two instance IDs, not one merged car blob,
panoptic output must assign every pixel exactly one segment ID.

That is why the same colored region can mean different things depending on the task. A purple region in semantic segmentation may mean "car pixels." In instance segmentation it must mean "this specific car." In panoptic segmentation it must live inside a full scene partition.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Runnable code 1def mask_iou(pred, target): assert len(pred) == len(target) intersection = 0 union = 0 for p_...python

def mask_iou(pred, target):
    assert len(pred) == len(target)
    intersection = 0
    union = 0
    for p_row, t_row in zip(pred, target):
        for p, t in zip(p_row, t_row):
            intersection += int(bool(p) and bool(t))
            union += int(bool(p) or bool(t))
    return 1.0 if union == 0 else intersection / union

car_target = [
    [0, 0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0, 0],
    [0, 1, 1, 1, 0, 0],
    [0, 1, 1, 1, 1, 0],
    [0, 0, 1, 1, 0, 0],
    [0, 0, 0, 0, 0, 0],
]

car_prediction = [
    [0, 0, 0, 0, 0, 0],
    [0, 0, 1, 0, 0, 0],
    [0, 1, 1, 1, 0, 0],
    [0, 1, 1, 1, 0, 0],
    [0, 0, 1, 1, 0, 0],
    [0, 0, 0, 0, 0, 0],
]

print(round(mask_iou(car_prediction, car_target), 3))

This witness ignores neural-network training. Its job is to expose the mask contract: a predicted region is judged by overlap with the target region, not only by the image-level class name.

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the Segmentation Mask Lab as a prediction-first check:

Pixel class: decide what semantic segmentation must output at each pixel.
Instance split: decide whether two same-class objects share one mask or receive separate IDs.
Panoptic rule: decide how stuff regions and thing instances combine into one scene map.

Before reveal, the pixel class, mask IoU, instance split, skip-detail note, and panoptic rule stay locked. After reveal, compare the same toy street scene across semantic, instance, U-Net-style detail recovery, and panoptic output. The goal is not to memorize architecture names; it is to see which output contract each route is built to satisfy.

Live Concept Demo

Explore Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Segmentation turns vision into pixel-level output: semantic classes for every pixel, separate thing instances, and panoptic maps that combine stuff regions with instance IDs.

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps should make visible.

Visual Inquiry

Make the image answer a mathematical question

Segmentation turns vision into pixel-level output: semantic classes for every pixel, separate thing instances, and panoptic maps that combine stuff regions with instance IDs.

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

paper · 2015Fully Convolutional Networks for Semantic SegmentationLong, Shelhamer, and Darrell

Primary support for pixel-to-pixel dense prediction, fully convolutional classifiers, upsampling, and skip/fusion context for semantic segmentation.

Open source

paper · 2015U-Net: Convolutional Networks for Biomedical Image SegmentationRonneberger, Fischer, and Brox

Primary support for encoder-decoder segmentation with skip connections that combine localization detail with high-level features.

Open source

paper · 2017Mask R-CNNHe, Gkioxari, Dollar, and Girshick

Primary support for instance segmentation with detected object boxes and a per-instance mask branch.

Open source

paper · 2018Panoptic SegmentationKirillov, He, Girshick, Rother, and Dollar

Primary support for the semantic/instance unification where every pixel receives a semantic label and thing pixels also receive instance IDs.

Open source

Claim Review

Segmentation turns vision into pixel-level output: semantic classes for every pixel, separate thing instances, and panoptic maps that combine stuff regions with instance IDs.

Status2 substantive reviews recorded

Claims without a substantive review badge still need exact source-support review.

Sources4 references

long-2015-fcn, ronneberger-2015-unet, he-2017-mask-rcnn, kirillov-2018-panoptic

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedSemantic segmentation predicts a class for each pixel; instance segmentation separates countable objects even when they share a class; panoptic segmentation assigns every pixel a semantic label while giving thing pixels unique instance IDs.Claim metadata: source checked

The references support dense per-pixel semantic outputs, per-instance mask prediction for detected things, and the panoptic requirement that stuff and thing pixels be handled in one scene-level labeling.

Sources: Fully Convolutional Networks for Semantic Segmentation, Mask R-CNN, Panoptic SegmentationThis page teaches the output contracts and basic mask metrics, not an exhaustive leaderboard, all segmentation architectures, medical-use validation, or production annotation policy.A bounded review summary is present; still check caveats and exact reference scope.

Checked FCN, Mask R-CNN, and Panoptic Segmentation for the output-contract distinction between per-pixel semantic classes, per-object instance masks, and the unified panoptic map. GPT Pro publication critique remains pending because the Oracle wrapper and remote Chrome port are unavailable on this workstation.

Reviewer: codex-local-primary-reference-audit; reviewed 2026-07-04

source-span-long-2015-fcn source-span-he-2017-mask-rcnn source-span-kirillov-2018-panoptic Equation 1 Runnable code 1 Interactive demo

Substantively reviewedFCN-style routes turn classifiers into dense predictors, U-Net-style routes recover detail through encoder-decoder skip connections, and Mask R-CNN-style routes attach a mask head to detected object instances.Claim metadata: source checked

The references support fully convolutional dense prediction, U-Net skip connections for localization, and Mask R-CNN's per-instance mask branch.

Sources: Fully Convolutional Networks for Semantic Segmentation, U-Net: Convolutional Networks for Biomedical Image Segmentation, Mask R-CNNModern segmentation systems often mix multi-scale features, transformers, pretraining, promptable masks, and task-specific losses; this artifact stays with the canonical route mechanics.A bounded review summary is present; still check caveats and exact reference scope.

Checked FCN, U-Net, and Mask R-CNN for dense prediction, skip-connected decoder structure, and per-instance mask heads. The local demo is a small mask-contract witness rather than an implementation of these architectures.

Reviewer: codex-local-primary-reference-audit; reviewed 2026-07-04

source-span-long-2015-fcn source-span-ronneberger-2015-unet source-span-he-2017-mask-rcnn Equation 2 Runnable code 1 Interactive demo

Source support candidates

paper 2015Fully Convolutional Networks for Semantic Segmentation

Primary support for pixel-to-pixel dense prediction, fully convolutional classifiers, upsampling, and skip/fusion context for semantic segmentation.

paper 2015U-Net: Convolutional Networks for Biomedical Image Segmentation

Primary support for encoder-decoder segmentation with skip connections that combine localization detail with high-level features.

paper 2017Mask R-CNN

Primary support for instance segmentation with detected object boxes and a per-instance mask branch.

paper 2018Panoptic Segmentation

Primary support for the semantic/instance unification where every pixel receives a semantic label and thing pixels also receive instance IDs.

Mechanism checks

Equation 1

P \in \mathbb R^{H\times W\times C}.

Equation 2

\mathcal L_{\mathrm{sem}} = -\frac{1}{HW}\sum_{h=1}^{H}\sum_{w=1}^{W} \log P_{h,w,y_{h,w}}.

Runnable code 1def mask_iou(pred, target): assert len(pred) == len(target) intersection = 0 union = 0 for p_...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Segmentation turns vision into pixel-level output: semantic classes for every pixel, separate thing instances, and panoptic maps that combine stuff regions with instance IDs.

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Grounded research drawerClose

ConceptSegmentation: FCN, U-Net, Mask R-CNN, and Panoptic MapsMachine Learning

Runnable code comparisonSegmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps runnable code 1def mask_iou(pred, target): assert len(pred) == len(target) intersection = 0 union = 0 for p_...Prediction before revealSegmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptMachine Learning

Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps

Attached question

What is the smallest example that makes Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays in this browser, attached to the selected learning item.

Draft noteNext action

No local draft saved.

Evidence to inspect

References to inspect: attached references on this page.
Definition, prerequisite, and contrast concept links
The equation or runnable code that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps Selected item key: recorded for copy. Context: Machine Learning Page anchor: recorded for copy. Open question: What is the smallest example that makes Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context

concept/concept-notebook/machine-learning/segmentation-fcn-unet-mask-rcnn
concept:machine-learning/segmentation-fcn-unet-mask-rcnn

Learning Map

Before / Now / Try / Next

BeforeCNN Shape Mechanics: Convolution, Padding, Stride, Pooling, and Channels (review)

NowIntuition -> Math -> Code -> Demo

TryManipulate one control and predict the visible change.

NextSemantic, Instance, and Panoptic Segmentation (review)

Intuitionready
Mathready
Codeready
Interactive Demoready

AI Companion

Ask beside the selected object

Segmentation turns vision into pixel-level output: semantic classes for every pixel, separate thing instances, and panoptic maps that combine stuff regions with instance IDs.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Machine Learning concept. Learning surface: Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps. What this page says: Segmentation turns vision into pixel-level output: semantic classes for every pixel, separate thing instances, and panoptic maps that combine stuff regions with instance IDs. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

Machine Learning

machine-learningcomputer-visionsegmentationdense-predictionmaskscnn

Prerequisites

CNN Shape Mechanics: Convolution, Padding, Stride, Pooling, and Channels (review)Image Classification Pipeline (review)Object Detection: R-CNN, YOLO, SSD, and DETR (review)

Leads To

Semantic, Instance, and Panoptic Segmentation (review)

CNN Architecture Families (review)Modern CNN Block Mechanics (review)Vision Transformers (review)Self-Supervised Vision: SimCLR, MoCo, DINO, and MAE (review)Semantic, Instance, and Panoptic Segmentation (review)

Within this domain