Combining Semantic and Geometric Features for Object Class Segmentation of Indoor Scenes

Abstract

Abstract-Scene understanding is a necessary prerequisite for robots acting autonomously in complex environments. Low-cost RGB-D cameras such as Microsoft Kinect enabled new methods for analyzing indoor scenes and are now ubiquitously used in indoor robotics. We investigate strategies for efficient pixelwise object class labeling of indoor scenes that combine both pretrained semantic features transferred from a large color image dataset and geometric features, computed relative to the room structures, including a novel distance-from-wall feature, which encodes the proximity of scene points to a detected major wall of the room. We evaluate our approach on the popular NYU v2 dataset. Several deep learning models are tested, which are designed to exploit different characteristics of the data. This includes feature learning with two different pooling sizes. Our results indicate that combining semantic and geometric features yields significantly improved results for the task of object class segmentation.

Semantic segmentation priors for object discovery

“…We use the semantic segmentation method of Husain et al [4], which is a feature learning approach similar to Eigen and Fergus [17] and Long et al [18]. Other approaches for semantic segmentation introduce hand-crafted features in their model such as gradient, colour, local binary pattern, depth gradient, spin, surface normals by Wu et al [19] and pixel value comparison and oriented gradients by Hermans et al [20].…”

“…Here, saliency is used to locate the objects, while colour and depth segmentations are used to define their precise boundaries. For semantic segmentation, we chose the deep convolutional neural network approach of Husain et al [4], which produces state-of-theart segmentation results. The semantic segmentation induces a partitioning of the superpixel set into semantic categories, to which the object discovery method can be applied separately.…”

“…2. Illustrating the top-down semantic segmentation as proposed by Husain et al [4]. Inputs are the colour image and the depth map.…”

“…We use saliency as a cue to locate the presence of objects, and colour and depth bottom-up segmentations are used to precisely delimit their boundaries. We improve this method by incorporating prior information on the category of the image regions using a semantic segmentation algorithm [4]. Husain et al [4] train a convolutional neural network to produce a pixel-wise classification of the image into four coarse categories: Floor, structures such as walls, furniture, and movable objects.…”

“…We improve this method by incorporating prior information on the category of the image regions using a semantic segmentation algorithm [4]. Husain et al [4] train a convolutional neural network to produce a pixel-wise classification of the image into four coarse categories: Floor, structures such as walls, furniture, and movable objects. The pixel-precise semantic maps are used to label the bottom-up segments for each category.…”