Towards Bounding-Box Free Panoptic Segmentation

Abstract

In this work we introduce a new bounding-box free network (BBFNet) for panoptic segmentation. Panoptic segmentation is an ideal problem for a bounding-box free approach as it already requires per-pixel semantic class labels. We use this observation to exploit class boundaries from an off-the-shelf semantic segmentation network and refine them to predict instance labels. Towards this goal BBFNet predicts coarse watershed levels and use it to detect large instance candidates where boundaries are well defined. For smaller instances, whose boundaries are less reliable, BBFNet also predicts instance centers by means of Hough voting followed by mean-shift to reliably detect small objects. A novel triplet loss network helps merging fragmented instances while refining boundary pixels. Our approach is distinct from previous works in panoptic segmentation that rely on a combination of a semantic segmentation network with a computationally costly instance segmentation network based on bounding boxes, such as Mask R-CNN, to guide the prediction of instance labels using a Mixture-of-Expert (MoE) approach. We benchmark our non-MoE method on Cityscapes and Microsoft COCO datasets and show competitive performance with other MoE based approaches while outperfroming exisiting non-proposal based approaches. We achieve this while been computationally more efficient in terms of number of parameters and FLOPs. Video results are provided here https://blog.slamcore.com/ reducing-the-cost-of-understanding

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation

“…SSAP [27] exploits the pixel-pair affinity pyramid [58] enabled by an efficient graph partition method [42]. BBFNet [7] obtains instance segmentation results by Watershed transform [79,4] and Hough-voting [5,47]. Recently, Panoptic-DeepLab [18], a simple, fast, and strong approach for bottom-up panoptic segmentation, employs a class-agnostic instance segmentation branch involving a simple instance center regression [41,77,61], coupled with DeepLab semantic segmentation outputs [12,14,15].…”

CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation

“…Then, the instance segments ('thing') and semantic segments ('stuff') [12] are fused by merging modules [52-54, 63, 70, 90, 92] to generate panoptic segmentation. Other proxy-based methods typically start with semantic segments [11,13,16] and group 'thing' pixels into instance segments with various proxy tasks, such as instance center regression [19,42,56,67,80,84,91], Watershed transform [4,8,82], Hough-voting [5,8,51], or pixel affinity [8,29,43,61,77]. DetectoRS [71] achieved the state-ofthe-art in this category with recursive feature pyramid and switchable atrous convolution.…”

MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers

“…Contrary to box-based approaches, box-free methods typically start with semantic segments [12,14,16]. Then, instance segments are obtained by grouping 'thing' pixels with various methods, such as instance center regression [44,86,70,100,20], Watershed transform [88,3,8], Hough-voting [4,53,8], or pixel affinity [45,66,81,30,8]. Recently, Axial-DeepLab [89] advanced the state-of-the-art by equipping Panoptic-DeepLab [21] with a fully axial-attention [35] backbone.…”