Fast and Compact Image Segmentation Using Instance Stixels

Abstract

State-of-the-art stixel methods fuse dense stereo disparity and semantic class information, e.g., from a Convolutional Neural Network (CNN), into a compact representation of driveable space, obstacles and background. However, they do not explicitly differentiate instances within the same semantic class. We investigate several ways to augment single-frame stixels with instance information, which can be extracted by a CNN from the RGB image input. As a result, our novel Instance Stixels method efficiently computes stixels that account for boundaries of individual objects, and represents instances as grouped stixels that express connectivity. Experiments on the Cityscapes dataset demonstrate that including instance information into the stixel computation itself, rather than as a post-processing step, increases the segmentation performance (i.e., Intersection over Union and Average Precision). This holds especially for overlapping objects of the same class. Furthermore, we show the superiority of our approach in terms of segmentation performance and computational efficiency compared to combining the separate outputs of Semantic Stixels and a state-of-the-art pixel-level CNN. We achieve processing throughput of 28 frames per second on average for 8 pixel wide stixels on images from the Cityscapes dataset at 1792 × 784 pixels. Our Instance Stixels software is made freely available for non-commercial research purposes.

Exploring Uncertainty-Based Self-Prompt for Test-Time Adaptation Semantic Segmentation in Remote Sensing Images

“…Building upon these works, [45] proposes a hierarchical and interpretable approach to autonomous driving, providing a multi-scale perspective for future research. FCIS [46] introduces an instance-level pixel inference method for more accurate contour detection, while [47] presents a scene-adaptive multi-scale semantic perception framework for 3D semantic segmentation, estimating spatial occupancy in the scene. MPCNet [48] proposed a multi-pool contextual segmentation network with high speed.…”

Detecting Darting Out Pedestrians With Occlusion Aware Sensor Fusion of Radar and Stereo Camera

“…Stixels are rectangular column-wise group of pixels based on disparity information with the goal of reducing the complexity of the stereo point cloud. Since the original publication [49], researchers have integrated class information [50] and later instance information into stixels [51]. The latter are referred to as Instance Stixels and could be a well suited input for an occlusion model because they follow the shape of an occluding car (both in depth and width/height) and are still computationally efficient to compute and process.…”

“…2, top. For both tasks, we use the Instance Stixel representation [51]. Stixels [49] are rectangular upright sticks in the 3D space, perpendicular to the estimated ground plane.…”

“…Stixels [49] are rectangular upright sticks in the 3D space, perpendicular to the estimated ground plane. With the extension of [51], each stixel has the following parameters: a 3D position of their bottom, a height, a class label (among others: car, bus, truck, person, sky) and an instance id. In this way, objects of interest (pedestrians and occluding vehicles) are represented by a loose set of stixels connected by their class and instance information.…”