Machine Learning

Semantic, Instance, and Panoptic Segmentation

Compare category maps, object masks, and panoptic scene partitions on the same toy image so the output contract is visible before the metric names arrive.

status: reviewimportance: importantdifficulty 4/5math: graduateread: 18mlive demo

Back to Machine Learning Next: Seq2seq and Encoder-Decoder Attention

Concept Structure

Semantic, Instance, and Panoptic Segmentation

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

1prerequisites

0next concepts

4related links

Learner Contract

What this page should let you do.

You are here becauseCompare category maps, object masks, and panoptic scene partitions on the same toy image so the output contract is visible before the metric names arrive.

This Machine Learning concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

Before thisSegmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps (review)

1 prerequisite listed; refresh them before leaning on the math or code.

Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps (review)

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextSeq2seq and Encoder-Decoder Attention

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Seq2seq and Encoder-Decoder Attention

Claim/source review status

Substantive review recorded

2/2 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.

Claims2/2 reviewed

Sources4 cited

Codeattached

Demolive

Reviewed2026-07-04

Updatedpage 2026-07-04

Learning item flow

4/4 sections readyAsk about this Research room

ConceptSemantic, Instance, and Panoptic SegmentationMachine Learning EquationSemantic, Instance, and Panoptic Segmentation equation 1Exact equation CodeSemantic, Instance, and Panoptic Segmentation runnable code 1Exact runnable code DemoSemantic, Instance, and Panoptic Segmentation interactive demoInteractive demo ClaimSemantic segmentation predicts category labels for pixels, instance s...Exact claim check SourceFully Convolutional Networks for Semantic SegmentationExact source

ConceptSemantic, Instance, and Panoptic SegmentationMachine Learning

4 sources attachedLocal snapshot ready

concept:machine-learning/semantic-instance-panoptic-segmentation

Codenearby Predictbefore reveal Roomitem handoff

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

The same street image can produce three correct-looking masks that mean different things.

Semantic segmentation answers:

Which category is this pixel?

Both cars can share one car color. Both people can share one person color. Road, sky, building, vegetation, and sidewalk are labeled as regions. The output is dense, but it does not count objects.

Instance segmentation answers:

Which countable object is this mask?

Car #1 and car #2 must be separate. Person #1 and person #2 must be separate. The output is about thing identity, so amorphous stuff such as road or sky may not be the focus.

Panoptic segmentation answers:

Which segment owns this pixel, and what category is that segment?

Every non-ignored pixel belongs to one segment. Stuff segments carry category labels. Thing segments carry category labels plus unique instance identities. This is why panoptic output is stricter than simply placing an instance mask on top of a semantic map: overlaps, holes, merged objects, and ignored pixels all matter.

The useful habit is to ask what contract the map is promising before asking whether it looks pretty.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1 $s:\Omega \to \mathcal C.$ Equation 2 $\{(c_k, M_k)\}_{k=1}^{K},$

Let $\Omega$ be the set of image pixels and $\mathcal C$ be the category set.

Semantic segmentation is a function:

s:\Omega \to \mathcal C.

It assigns one category to each scored pixel. If two cars are present, both can map to the same category value.

Instance segmentation returns a set of thing masks:

\{(c_k, M_k)\}_{k=1}^{K},

where $c_k$ is a thing category and $M_k \subseteq \Omega$ is the mask for one object instance.

Panoptic segmentation combines both views. Each non-ignored pixel receives exactly one segment ID:

p:\Omega \setminus V \to \{1,\ldots,N\},

and each segment ID has metadata:

m_j = (\mathrm{category}_j, \mathrm{isThing}_j, \mathrm{instanceId}_j).

The void set $V$ is excluded from scoring. It is not another stuff category.

For a simplified single-category panoptic-quality witness, suppose matched segment IoUs sum to $I$ , with true positives $TP$ , false positives $FP$ , and false negatives $FN$ :

PQ = \frac{I}{TP + \frac{1}{2}FP + \frac{1}{2}FN}.

This can be decomposed into:

SQ = \frac{I}{TP}, \qquad RQ = \frac{TP}{TP + \frac{1}{2}FP + \frac{1}{2}FN}, \qquad PQ = SQ \cdot RQ.

The segmentation part asks whether matched masks overlap well. The recognition part penalizes extra segments and missed segments. That is the heart of the panoptic contract: classify the pixels, preserve object identities, and cover the scene without contradictory ownership.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Runnable code 1def pq_witness(matched_ious, false_positives, false_negatives): true_positives = len(matched_...python

def pq_witness(matched_ious, false_positives, false_negatives):
    true_positives = len(matched_ious)
    if true_positives == 0:
        return {"sq": 0.0, "rq": 0.0, "pq": 0.0}

    iou_sum = sum(matched_ious)
    denominator = true_positives + 0.5 * false_positives + 0.5 * false_negatives
    sq = iou_sum / true_positives
    rq = true_positives / denominator
    pq = iou_sum / denominator
    return {"sq": sq, "rq": rq, "pq": pq}


car_result = pq_witness(
    matched_ious=[0.86, 0.78],
    false_positives=1,
    false_negatives=0,
)

print(round(car_result["sq"], 3), round(car_result["rq"], 3), round(car_result["pq"], 3))

This witness is deliberately tiny. Its job is to make the false-positive and false-negative penalties visible before a learner meets a full benchmark implementation.

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the Stuff/Thing Census Lab as a prediction-first comparison:

Semantic merge: decide which map labels every scored pixel by category while merging same-class objects.
Thing identity: decide which map separates countable objects but does not promise a full stuff map.
Panoptic cover: decide which map gives the whole scene a non-overlapping stuff-plus-thing partition.
Void boundary: decide how ignored pixels should be treated when checking coverage.

Before reveal, the category count, thing IDs, coverage, and PQ-style arithmetic stay locked. After reveal, compare the same toy street scene under all three output contracts.

Live Concept Demo

Explore Semantic, Instance, and Panoptic Segmentation

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Semantic, Instance, and Panoptic Segmentation should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Compare category maps, object masks, and panoptic scene partitions on the same toy image so the output contract is visible before the metric names arrive.

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Semantic, Instance, and Panoptic Segmentation should make visible.

Visual Inquiry

Make the image answer a mathematical question

Compare category maps, object masks, and panoptic scene partitions on the same toy image so the output contract is visible before the metric names arrive.

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Semantic, Instance, and Panoptic Segmentation easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

paper · 2015Fully Convolutional Networks for Semantic SegmentationLong, Shelhamer, and Darrell

Primary support for semantic segmentation as pixel-to-pixel class prediction.

Open source

paper · 2017Mask R-CNNHe, Gkioxari, Dollar, and Girshick

Primary support for instance segmentation as per-object mask prediction attached to detected things.

Open source

paper · 2018Panoptic SegmentationKirillov, He, Girshick, Rother, and Dollar

Primary support for unifying semantic and instance segmentation, plus PQ-style stuff/thing evaluation.

Open source

reference · 2018COCO Panoptic APICOCO Dataset

Reference support for COCO-style thing/stuff category flags, void handling, conversion utilities, and panoptic quality computation.

Open source

Claim Review

Compare category maps, object masks, and panoptic scene partitions on the same toy image so the output contract is visible before the metric names arrive.

Status2 substantive reviews recorded

Claims without a substantive review badge still need exact source-support review.

Sources4 references

long-2015-fcn, he-2017-mask-rcnn, kirillov-2018-panoptic, coco-panopticapi

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedSemantic segmentation predicts category labels for pixels, instance segmentation separates countable object masks, and panoptic segmentation assigns one full-scene segment identity to each non-ignored pixel while preserving category metadata.Claim metadata: source checked

The references support dense category prediction, instance mask prediction for detected things, panoptic full-scene partitions, and thing/stuff category flags.

Sources: Fully Convolutional Networks for Semantic Segmentation, Mask R-CNN, Panoptic Segmentation, COCO Panoptic APIThis artifact teaches output contracts and evaluation shape, not all segmentation architectures, all annotation policies, or production dataset governance.A bounded review summary is present; still check caveats and exact reference scope.

Checked semantic, instance, and panoptic references for the category-only, object-identity, and unified full-scene output contracts. GPT Pro publication critique remains pending because the Oracle wrapper and remote Chrome port are unavailable on this workstation.

Reviewer: codex-local-primary-reference-audit; reviewed 2026-07-04

Equation 1 Runnable code 1 Interactive demo

Substantively reviewedA simplified panoptic-quality witness can be explained as matched segment IoU divided by true positives plus half false positives plus half false negatives, with separate recognition and segmentation factors.Claim metadata: source checked

The references support PQ as a unified metric over stuff and thing categories and the implementation shape used here for a small deterministic witness.

Sources: Panoptic Segmentation, COCO Panoptic APIThe demo is intentionally small. It does not implement full dataset evaluation, crowd handling, category remapping, or all COCO panoptic file-format details.A bounded review summary is present; still check caveats and exact reference scope.

Checked the panoptic paper and COCO Panoptic API implementation for the PQ, SQ, RQ decomposition and the matched-segment threshold shape used by the teaching witness.

Reviewer: codex-local-primary-reference-audit; reviewed 2026-07-04

Equation 2 Runnable code 1 Interactive demo

Source support candidates

paper 2015Fully Convolutional Networks for Semantic Segmentation

Primary support for semantic segmentation as pixel-to-pixel class prediction.

paper 2017Mask R-CNN

Primary support for instance segmentation as per-object mask prediction attached to detected things.

paper 2018Panoptic Segmentation

Primary support for unifying semantic and instance segmentation, plus PQ-style stuff/thing evaluation.

reference 2018COCO Panoptic API

Reference support for COCO-style thing/stuff category flags, void handling, conversion utilities, and panoptic quality computation.

Mechanism checks

Equation 1

s:\Omega \to \mathcal C.

Equation 2

\{(c_k, M_k)\}_{k=1}^{K},

Runnable code 1def pq_witness(matched_ious, false_positives, false_negatives): true_positives = len(matched_...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Compare category maps, object masks, and panoptic scene partitions on the same toy image so the output contract is visible before the metric names arrive.

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in Semantic, Instance, and Panoptic Segmentation.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Grounded research drawerClose

ConceptSemantic, Instance, and Panoptic SegmentationMachine Learning

Runnable code comparisonSemantic, Instance, and Panoptic Segmentation runnable code 1def pq_witness(matched_ious, false_positives, false_negatives): true_positives = len(matched_...Prediction before revealSemantic, Instance, and Panoptic Segmentation interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes Semantic, Instance, and Panoptic Segmentation click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptMachine Learning

Semantic, Instance, and Panoptic Segmentation

Attached question

What is the smallest example that makes Semantic, Instance, and Panoptic Segmentation click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays in this browser, attached to the selected learning item.

Draft noteNext action

No local draft saved.

Evidence to inspect

References to inspect: attached references on this page.
Definition, prerequisite, and contrast concept links
The equation or runnable code that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Semantic, Instance, and Panoptic Segmentation Selected item key: recorded for copy. Context: Machine Learning Page anchor: recorded for copy. Open question: What is the smallest example that makes Semantic, Instance, and Panoptic Segmentation click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context

concept/concept-notebook/machine-learning/semantic-instance-panoptic-segmentation
concept:machine-learning/semantic-instance-panoptic-segmentation

Learning Map

Before / Now / Try / Next

BeforeSegmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps (review)

NowIntuition -> Math -> Code -> Demo

TryManipulate one control and predict the visible change.

NextSeq2seq and Encoder-Decoder Attention

Intuitionready
Mathready
Codeready
Interactive Demoready

AI Companion

Ask beside the selected object

Compare category maps, object masks, and panoptic scene partitions on the same toy image so the output contract is visible before the metric names arrive.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Machine Learning concept. Learning surface: Semantic, Instance, and Panoptic Segmentation. What this page says: Compare category maps, object masks, and panoptic scene partitions on the same toy image so the output contract is visible before the metric names arrive. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

Machine Learning

machine-learningcomputer-visionsegmentationsemantic-segmentationinstance-segmentationpanoptic-segmentationevaluation

Prerequisites

Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps (review)

Segmentation: FCN, U-Net, Mask R-CNN, and Panoptic Maps (review)Object Detection: R-CNN, YOLO, SSD, and DETR (review)Image Classification Pipeline (review)Vision Transformers (review)

Within this domain