ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors

Abstract

Instance segmentation aims to detect and segment individual objects in a scene. Most existing methods rely on precise mask annotations of every category. However, it is difficult and costly to segment objects in novel categories because a large number of mask annotations is required. We introduce ShapeMask, which learns the intermediate concept of object shape to address the problem of generalization in instance segmentation to novel categories. Shape-Mask starts with a bounding box detection and gradually refines it by first estimating the shape of the detected object through a collection of shape priors. Next, ShapeMask refines the coarse shape into an instance level mask by learning instance embeddings. The shape priors provide a strong cue for object-like prediction, and the instance embeddings model the instance specific appearance information. Shape-Mask significantly outperforms the state-of-the-art by 6.4 and 3.8 AP when learning across categories, and obtains competitive performance in the fully supervised setting. It is also robust to inaccurate detections, decreased model capacity, and small training data. Moreover, it runs efficiently with 150ms inference time and trains within 11 hours on TPUs. With a larger backbone model, ShapeMask increases the gap with state-of-the-art to 9.4 and 6.2 AP across categories. Code will be released.

ACFNet: Attentional Class Feature Network for Semantic Segmentation

“…There are a lot of successful applications of using coarse-to-fine approaches, such as face detection [13], shape detection [1], face alignment [49] and optical flow [3]. Some existing segmentation networks [19,50,36,21] also adopt coarse-to-fine strategy. Islam et al [19] combined high resolution features and coarse segmentation result of low resolution features to get a finer segmentation result.…”

Learning to Detect Every Thing in an Open World

“…However, from a methodological perspective, open-world object detection and segmentation remain understudied despite the importance of the task. Hu et al and Kuo et al [24,31] proposed approaches for predicting masks of various objects, but they require bounding boxes from classes of interest. Jaiswal et al [26] trained a detector in an adversarial manner to learn class-invariant objectness.…”

Mask Transfiner for High-Quality Instance Segmentation

“…The follow-up works [7,12,24,32,33] also contribute to the family of Mask R-CNN models. One-stage methods [5,8,26,27] and kernel-based method [48], such as PolarMask [44], YOLOACT [1], and SOLO [40,41] remove the proposal generation and feature re-pooling steps, achieving comparable results with higher efficiency.…”