Image Segmentation Models Cheat Sheet

Updated 2026-05-21

Next Topic: Imitation Learning and Learning from Demonstrations Cheat Sheet

🧠Study flashcards on this topic82 cards · spaced repetition→

Image segmentation is the computer vision task of partitioning every pixel of an image into a meaningful category — it sits at the heart of autonomous driving, medical imaging, and robotics. Unlike object detection, which draws boxes, segmentation produces pixel-precise masks, making the choice of model architecture directly tied to whether you need class labels (semantic), individual object identities (instance), or both (panoptic). The field has evolved from hand-crafted FCNs through encoder-decoder CNNs (U-Net, DeepLabV3+) and two-stage detectors (Mask R-CNN) to today's foundation models like SAM and SAM 2; the critical insight is that no single model dominates all tasks — trade-offs between accuracy, speed, and the type of segmentation required always drive the selection.

What This Cheat Sheet Covers

This topic spans 14 focused tables and 93 indexed concepts, 82 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Segmentation Task TypesTable 2: U-Net ArchitectureTable 3: Mask R-CNN ArchitectureTable 4: DeepLabV3+ ArchitectureTable 5: SAM — Segment Anything ModelTable 6: SAM 2 — Video and Image SegmentationTable 7: HQ-SAM — High-Quality Segmentation AdapterTable 8: PSPNet and SegFormer — Context Aggregation ArchitecturesTable 9: YOLACT — Real-Time Instance SegmentationTable 10: Evaluation MetricsTable 11: Dataset FormatsTable 12: Libraries and FrameworksTable 13: Fine-Tuning and Adaptation StrategiesTable 14: Common Pitfalls and Best Practices

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Segmentation Task Types

Segmentation divides into three distinct problem formulations that differ in what they require a model to know about each pixel. Understanding which task you are solving is the first decision, because it determines which architectures, metrics, and dataset formats are appropriate.

Type	Example	Description
Semantic segmentation	Every car pixel → class `car`; every sky pixel → class `sky`	• Assigns a single class label to every pixel • multiple objects of the same class share one label — no instance distinction • Handles "stuff" (amorphous regions) well
Instance segmentation	Person 1 → mask ID 1; person 2 → mask ID 2	• Detects and delineates each countable object ("things") individually, even within the same class • assigns unique instance IDs
Panoptic segmentation	Every pixel gets both a class label and an instance ID	• Unifies semantic and instance segmentation: every pixel receives a class label; "things" also receive unique instance IDs. • Defined as each pixel having a semantic label + optional instance ID • Instance wins on overlap for "things"
Things vs. Stuff distinction	Things: person, car, animal; Stuff: sky, road, grass	• Things are countable, separable objects (handled by instance seg) • Stuff is amorphous background regions (handled by semantic seg). Panoptic covers both in one unified output

Table 2: U-Net Architecture

U-Net (Ronneberger et al., 2015) remains the dominant architecture for medical and scientific image segmentation because it produces accurate masks from very small annotated datasets. The symmetric encoder-decoder shape with skip connections is the template that dozens of later models (UNet++, Attention U-Net, TransUNet) build upon.

Component	Example	Description
Encoder (contracting path)	Conv3×3 → ReLU → Conv3×3 → ReLU → MaxPool2×2; feature channels: 64→128→256→512→1024	• Progressively halves spatial resolution and doubles channels at each level via repeated 3×3 convolutions + 2×2 max-pooling • captures context at increasing abstraction

The rest of this section is part of the full cheat sheet

Table 3: Mask R-CNN Architecture

Mask R-CNN (He et al., 2017) is the canonical two-stage instance segmentation model: a region proposal stage identifies candidate objects, and a parallel mask head generates a pixel-wise binary mask for each detected instance. It extended Faster R-CNN by adding a third output branch without disrupting the existing detection pipeline.

Component	Example	Description
Backbone + FPN	ResNet-50 or ResNet-101 + Feature Pyramid Network	• Pre-trained CNN extracts features • FPN builds a multi-scale feature pyramid (top-down + lateral connections) enabling detection of objects at all scales

The rest of this section is part of the full cheat sheet

Table 4: DeepLabV3+ Architecture

DeepLabV3+ (Chen et al., 2018) is the most widely used semantic segmentation architecture for scenes with objects at multiple scales. Its combination of atrous convolutions (dilated convolutions), Atrous Spatial Pyramid Pooling (ASPP), and a lightweight decoder to refine boundaries produces state-of-the-art results on Pascal VOC and Cityscapes while remaining trainable on a single workstation.

Component	Example	Description
Atrous (dilated) convolution	Conv3×3 with dilation rate `r=6`; inserts `r-1` zeros between filter weights	• Expands the receptive field without increasing parameters or reducing resolution • rate `r` controls spacing between filter weights

The rest of this section is part of the full cheat sheet

Table 1: Segmentation Task Types

Type	Example	Description
Semantic segmentation	Every car pixel → class `car`; every sky pixel → class `sky`	• Assigns a single class label to every pixel • multiple objects of the same class share one label — no instance distinction • Handles "stuff" (amorphous regions) well
Instance segmentation	Person 1 → mask ID 1; person 2 → mask ID 2	• Detects and delineates each countable object ("things") individually, even within the same class • assigns unique instance IDs
Panoptic segmentation	Every pixel gets both a class label and an instance ID	• Unifies semantic and instance segmentation: every pixel receives a class label; "things" also receive unique instance IDs. • Defined as each pixel having a semantic label + optional instance ID • Instance wins on overlap for "things"
Things vs. Stuff distinction	Things: person, car, animal; Stuff: sky, road, grass	• Things are countable, separable objects (handled by instance seg) • Stuff is amorphous background regions (handled by semantic seg). Panoptic covers both in one unified output

Table 2: U-Net Architecture

Component	Example	Description
Encoder (contracting path)	Conv3×3 → ReLU → Conv3×3 → ReLU → MaxPool2×2; feature channels: 64→128→256→512→1024	• Progressively halves spatial resolution and doubles channels at each level via repeated 3×3 convolutions + 2×2 max-pooling • captures context at increasing abstraction

Table 3: Mask R-CNN Architecture

Component	Example	Description
Backbone + FPN	ResNet-50 or ResNet-101 + Feature Pyramid Network	• Pre-trained CNN extracts features • FPN builds a multi-scale feature pyramid (top-down + lateral connections) enabling detection of objects at all scales

Table 4: DeepLabV3+ Architecture

Component	Example	Description
Atrous (dilated) convolution	Conv3×3 with dilation rate `r=6`; inserts `r-1` zeros between filter weights	• Expands the receptive field without increasing parameters or reducing resolution • rate `r` controls spacing between filter weights