Single-shot Path Integrated Panoptic Segmentation

Abstract

Panoptic segmentation, which is a novel task of unifying instance segmentation and semantic segmentation, has attracted a lot of attention lately. However, most of the previous methods are composed of multiple pathways with each pathway specialized to a designated segmentation task. In this paper, we propose to resolve panoptic segmentation in single-shot by integrating the execution flows. With the integrated pathway, a unified feature map called Panoptic-Feature is generated, which includes the information of both things and stuffs. Panoptic-Feature becomes more sophisticated by auxiliary problems that guide to cluster pixels that belong to the same instance and differentiate between objects of different classes. A collection of convolutional filters, where each filter represents either a thing or stuff, is applied to Panoptic-Feature at once, materializing the single-shot panoptic segmentation. Taking the advantages of both top-down and bottom-up approaches, our method, named SPINet, enjoys high efficiency and accuracy on major panoptic segmentation benchmarks: COCO and Cityscapes.

Backbones-Review: Feature Extraction Networks for Deep Learning and Deep Reinforcement Learning Approaches

“…CSRNet [60] (2018) VGG-16 Axial-DeepLab [136] DeepLab SPN [61] (2019) VGG-16 BANet [137] ResNet-50 DENet [63] VGG-16 VSPNet [138] ResNet-50 CANNet [64] VGG-16 BGRNet [139] ResNet50 SCAR [65] VGG-16 SpatialFlow [140] ResNet50 ADNet [69] VGG-16 Weber et al [141] ResNet50 ADSCNet [69] VGG-16 AUNet [142] ResNet50 ASNet [70] VGG-16 OANet [143] ResNet50 SCNet [68] VGG-16 SPINet [144] ResNet50 BL [66] VGG-19 Son et al [145] ResNet-50 MobileCount [62] MobileNet-V2 SOGNet [146] ResNet101 SFCN [67] ResNet-101 DR1Mask [147] ResNet101…”

“…Also using ResNet-50, an end-to-end occlusion-aware network (OANet) is introduced in [143] to perform a panoptic segmentation, which uses a single network to predict instance and semantic segmentation. Without unifying the instance and semantic segmentation to get the panoptic segmentation, Hwang et al [144] exploited the blocks and pathways integration that allow unified feature maps that represent the final panoptic outcome. Finally, in [145], aiming at visualizing the hidden enemies in a scene, a panoptic segmentation method is proposed.…”

Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning

“…Seamless-Scene-Segmentation [12] directly incorporates an instance segmentation head following Mask R-CNN and a light-weight DeepLab-style semantic segmentation head. Most recently, there are single-path architectures without using separate branches like DETR [16], Panoptic FCN [17] and SPINet [18]. In this work, we build on seamless segmentation in an efficient setting, considering that fast responses are critical for autonomous driving systems.…”