Semantic Instance Labeling Leveraging Hierarchical Segmentation

Abstract

Most of the approaches for indoor RGBD semantic labeling focus on using pixels or superpixels to train a classifier. In this paper, we implement a higher level segmentation using a hierarchy of superpixels to obtain a better segmentation for training our classifier. By focusing on meaningful segments that conform more directly to objects, regardless of size, we train a random forest of decision trees as a classifier using simple features such as the 3D size, LAB color histogram, width, height, and shape as specified by a histogram of surface normals. We test our method on the NYU V2 depth dataset, a challenging dataset of cluttered indoor environments. Our experiments using the NYU V2 depth dataset show that our method achieves state of the art results on both a general semantic labeling introduced by the dataset (floor, structure, furniture, and objects) and a more object specific semantic labeling. We show that training a classifier on a segmentation from a hierarchy of super pixels yields better results than training directly on super pixels, patches, or pixels as in previous work.

Segmentation and semantic labelling of RGBD data with convolutional neural networks and surface fitting

“…Notice that the authors of [26] achieve very high performances by exploiting a much more complex deep learning architecture. In any case, our method is the one that gets closer to it, while even the very recent methods of [39, 25] have lower performances than ours.…”

“…In order to evaluate the accuracy of this labelling, we compared it with some competing approaches on the NYUD2 test set. The compared state-of-the-art approaches are the methods of [23] that uses a multi-scale CNN, of [39] that uses a hierarchy of super-pixels to train a random forest classifier, of [27] that uses deep learning to extract super-pixels features, of [25] exploiting two different CNNs, of [20] using random forest and CRFs and finally [26] using a multi-scale deep learning architecture. Table 2 reports the results: two different metrics have been considered, the per-pixel accuracy, counting the percentage of correctly classified pixels and the average class accuracy, obtained by computing the percentage of correctly classified pixels for each class independently and averaging the values.…”

“…Couprie et al [23] 52.4 36.2 Hickson et al [39] 53.0 47.6 A. Wang et al [27] 46.3 42.2 J. Wang et al [25] 54.8 52.7 A. Hermans et al [20] 54.2 48.0 D. Eigen et al [26] 75.4 66.9 proposed method (CNN output) 64.4 51.7 proposed method (with segmentation) 67.2 54.4 Table 3 Average accuracy for each of the 13 classes on the test set of the NYUD2 dataset for the proposed approach (the unknown class has not been considered consistently with the evaluation of all the compared approaches) Class Accuracy (CNN), (%) Accuracy (with segmentation), (%) Accuracy improvement, (%…”

“…In order to evaluate the accuracy of this labelling, we compared it with some competing approaches on the NYUD2 test set. The compared state-of-the-art approaches are the methods of [23] that uses a multi-scale CNN, of [39] that uses a hierarchy of super-pixels to train a random forest classifier, of [27] that uses deep learning to extract super-pixels features, of [25] exploiting two different CNNs, of [20] using random forest and CRFs and finally [26] using a multi-scale deep learning architecture.…”

Higher-order class-specific priors for semantic segmentation of 3D outdoor scenes

“…For example, Lin et al [18] proposed a 3D scene recovery system, in which objects in different categories are reconstructed with domain-specific knowledge based on the results of semantic segmentation. Unlike the RGB-D data used by the previous works [3,7,8,19], outdoor LIDAR data for semantic segmentation lacks the color and texture information. Meanwhile, accurate semantic segmentation of 3D point clouds is a challenging task due to occlusions caused by obstructions, varying point density caused by different distances of objects from the sensor and complex object structures appearing in 3D point clouds.…”