Pose2Seg: Detection Free Human Instance Segmentation

Abstract

The standard approach to image instance segmentation is to perform the object detection first, and then segment the object from the detection bounding-box. More recently, deep learning methods like Mask R-CNN [14] perform them jointly. However, little research takes into account the uniqueness of the "human" category, which can be well defined by the pose skeleton. Moreover, the human pose skeleton can be used to better distinguish instances with heavy occlusion than using bounding-boxes. In this paper, we present a brand new pose-based instance segmentation framework 1 for humans which separates instances based on human pose, rather than proposal region detection. We demonstrate that our pose-based framework can achieve better accuracy than the state-of-art detectionbased approach on the human instance segmentation problem, and can moreover better handle occlusion. Furthermore, there are few public datasets containing many heavily occluded humans along with comprehensive annotations, which makes this a challenging problem seldom noticed by researchers. Therefore, in this paper we introduce a new benchmark "Occluded Human (OCHuman)" 2 , which focuses on occluded humans with comprehensive annotations including bounding-box, human pose and instance masks. This dataset contains 8110 detailed annotated human instances within 4731 images. With an average 0.67 Max-IoU for each person, OCHuman is the most complex and challenging dataset related to human instance segmentation. Through this dataset, we want to emphasize occlusion as a challenging problem for researchers to study.

PoSeg: Pose-Aware Refinement Network for Human Instance Segmentation

“…However, as it highly relies on the performance of pose grouping, its overall performance is only on par with generic segmentation methods such as Mask R-CNN. A recent approach, Pose2Seg [1], also uses estimated human pose to generate instance proposals, which suffers from low recall rate in comparison to detection-based proposals. In addition, their segmentation is predicated based on the estimated poses, which leads to propagation of the grouping errors of poses to segmentation.…”

“…Solving this task plays an important role in scene understanding, evidenced by recent surge of interest in computer vision community [3]- [6]. Among all the object categories considered, human instance has attracted significant attention due to its wide range of real-world applications, such as human-robot interaction [7], human behavior analysis [8], and autonomous driving [9], as well as the unique challenges in human segmentation for general scenes [1].…”

“…A key reason is that instance grouping is ambiguous given a box proposal when multiple object instances are heavily overlapped inside the box region. To tackle this problem, recently Zhang et al [1] propose a bottom-up approach that utilizes estimated human poses to generate instance proposals and then performs human segmentation for each proposal. It leverages the grouping of sparse keypoints as an intermediate stage to mitigate the problem of ambiguity in dense segmentation.…”

“…In this work, we propose a new pose-aware approach to human instance segmentation in order to address the aforementioned limitations. In contrast to previous pose-aware methods [1], [2] which estimate instance segmentation on top of predicted poses, our method integrates both top-down and bottom-up cues of an instance: we adopt the detection results as human proposals to achieve high recall rate, and jointly estimate human pose and instance segmentation in each detection proposal. In this way, we are able to integrate proposal features and inferred pose cues in a principled manner, preventing error propagation from pose-based grouping to final segmentation.…”

“…We evaluate our approach on two public multi-person benchmarks, OCHuman [1] and COCOPersons [12] dataset. The results show that our method achieves the new state of the art on both datasets, evidencing the effectiveness of our proposed strategy.…”