One Label Per Pixel

Three cats stand shoulder to shoulder in a street photo. Two models are available:

  • Model A emits one mask covering every cat pixel, labelled cat.
  • Model B emits three masks, one per cat, each with its own confidence score, and the masks may overlap where the animals touch.

The product needs a single scene layer in which every pixel carries exactly one class label and one instance id, including road and sky pixels, with no overlaps anywhere. Which assessment is correct?

A

The two differ only in the training loss, so swapping per-pixel cross-entropy for a per-instance mask loss turns A into B without changing the output format.

B

A is semantic segmentation and B is instance segmentation, and neither meets the requirement: B allows overlapping masks and covers only countable things, so the product needs panoptic output, one class plus one instance id per pixel, with the id ignored for stuff such as road and sky.

C

B already meets it, since instance segmentation is a strict generalisation of semantic segmentation: merging the three masks recovers a per-pixel class map, and the leftover pixels are simply background.

D

A already meets it, since every pixel has a class: instance ids can be recovered afterwards by taking connected components of the cat mask.