SegSort: Segmentation by Discriminative Sorting of Segments

Abstract

Almost all existing deep learning approaches for semantic segmentation tackle this task as a pixel-wise classification problem. Yet humans understand a scene not in terms of pixels, but by decomposing it into perceptual groups and structures that are the basic building blocks of recognition. This motivates us to propose an end-to-end pixelwise metric learning approach that mimics this process. In our approach, the optimal visual representation determines the right segmentation within individual images and associates segments with the same semantic classes across images. The core visual learning problem is therefore to maximize the similarity within segments and minimize the similarity between segments. Given a model trained this way, inference is performed consistently by extracting pixel-wise embeddings and clustering, with the semantic label determined by the majority vote of its nearest neighbors from an annotated set. As a result, we present the SegSort, as a first attempt using deep learning for unsupervised semantic segmentation, achieving 76% performance of its supervised counterpart. When supervision is available, Seg-Sort shows consistent improvements over conventional approaches based on pixel-wise softmax training. Additionally, our approach produces more precise boundaries and consistent region predictions. The proposed SegSort further produces an interpretable result, as each choice of label can be easily understood from the retrieved nearest segments.

Exploring Cross-Image Pixel Contrast for Semantic Segmentation

“…Although a few prior methods address the idea of metric learning in semantic segmentation, they only account for the local content from objects [29] or instances [16,1,22,42]. It is worth noting [37] also explores cross-image information of training data, i.e., leverage perceptual pixel groups for nonparametric pixel classification. Due to its clustering based metric learning strategy, [37] needs to retrieve extra labeled data for inference.…”

“…It is worth noting [37] also explores cross-image information of training data, i.e., leverage perceptual pixel groups for nonparametric pixel classification. Due to its clustering based metric learning strategy, [37] needs to retrieve extra labeled data for inference. Differently, our core idea, i.e., exploit inter-image pixel-to-pixel similarity to enforce global constraints on the embedding space, is conceptually novel and rarely explored before.…”

“…Basically, these segmentation models (excluding [37]) utilize deep architectures to project image pixels into a highly non-linear embedding space (Fig. 1(c)).…”

Novel Class Discovery in Semantic Segmentation

“…SSL [4,41] concentrates on efficiently utilizing scarce labeled and enormous unlabeled data of the same label space. USL [13,31] rid of heavy annotation cost. WSSS [40] [36] further improve the pipeline by dual ranking statistics and mutual knowledge distillation.…”

“…Such data can be easily collected but difficult to annotate. USL [13,31] is also introduced to mitigate the annotation cost. Due to the complexity of unlabeled data, USL cannot achieve satisfactory results without any prior knowledge.…”