Combinatorial Optimization for Panoptic Segmentation: A Fully Differentiable Approach

Abstract

We propose an end-to-end trainable architecture for simultaneous semantic and instance segmentation (a.k.a. panoptic segmentation) consisting of a convolutional neural network and an asymmetric multiway cut problem solver. The latter solves a combinatorial optimization problem that elegantly incorporates semantic and boundary predictions to produce a panoptic labeling. Our formulation allows to directly maximize a smooth surrogate of the panoptic quality metric by backpropagating the gradient through the optimization problem. Experimental evaluation shows improvement of end-to-end learning w.r.t. comparable approaches on Cityscapes and COCO datasets. Overall, our approach shows the utility of using combinatorial optimization in tandem with deep learning in a challenging large scale real-world problem and showcases benefits and insights into training such an architecture end-to-end.Preprint. Under review.

RAMA: A Rapid Multicut Algorithm on GPU

“…Cityscapes Unsupervised image segmentation on 59 high resolution images (2048 × 1024) taken from the validation set [17]. Conversion to multicut instances is done by computing the edge affinities produced by [2] on a grid graph with 4-connectivity and additional coarsely sampled longer range edges. Each instance contains approximately 2 million nodes and 9 million edges.…”

“…The multicut problem and its extensions such as higher order multicut [27,32], lifted multicut [30], (asymmetric) multiway cut [14,36], lifted disjoint paths [21] and joint multicut and node labeling [41] have found numerous applications in machine learning, computer vision, biomedical image analysis, data mining and beyond. Examples include unsupervised image segmentation [4,5,7,56], instanceseparating semantic segmentation [2,33], multiple object tracking [21,51], cell tracking [25], articulated human body pose estimation [22], motion segmentation [31], image and mesh segmentation [30], connectomics [6,13,47] and many more.…”

“…In other scenarios very fast running times are essential, e.g. when multicut is used in end-to-end training [2,48]. Hence, the need for parallelization arises, preferably on GPUs.…”