Real-Time Progressive 3D Semantic Segmentation for Indoor Scenes

Abstract

The widespread adoption of autonomous systems such as drones and assistant robots has created a need for realtime high-quality semantic scene segmentation. In this paper, we propose an efficient yet robust technique for on-the-fly dense reconstruction and semantic segmentation of 3D indoor scenes. To guarantee (near) real-time performance, our method is built atop an efficient super-voxel clustering method and a conditional random field with higher-order constraints from structural and object cues, enabling progressive dense semantic segmentation without any precomputation. We extensively evaluate our method on different indoor scenes including kitchens, offices, and bedrooms in the SceneNN and ScanNet datasets and show that our technique consistently produces state-of-the-art segmentation results in both qualitative and quantitative experiments.

Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery

“…To compare against previous work in [5], we evaluate the 3D segmentation accuracy of the proposed dense object-level semantic mapping framework on real-world indoor scans from the SceneNN [8] dataset, improving over the baseline for most of the evaluated scenes. A sample inventory of object models discovered in these scenes is shown to contain recognized, semantically annotated elements, as well as newly discovered, previously unseen objects.…”

“…In their work, Pham et al [5] report instance-level 3D segmentation accuracy results for the NYUDv2 40 class task, which includes commonly-encountered indoor object classes, as well as structural, non-object categories, such as wall, window, door, floor, and ceiling. This set of classes is wellsuited for semantic segmentation tasks in which the goal is to classify and label every single element, either voxel of surfel, of the 3D scene.…”

“…This set of classes is wellsuited for semantic segmentation tasks in which the goal is to classify and label every single element, either voxel of surfel, of the 3D scene. Indeed, the approach in [5] initially employs a purely semantic segmentation strategy, and later clusters the semantically annotated scene into individual instances. However, a set of classes which includes non-object categories does not apply to the object-based segmentation approach proposed in this work.…”

JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds With Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields

“…Per-vertex labels are obtained by back-projecting and fusing 2D predictions from colour or RGB-D images onto 3D space. Predictions on 2D can be done via classifiers, e.g., random forests [14,36,46,42], or deep neural networks [27,49,30]. Such techniques can be implemented in tandem with 3D scene reconstruction, creating a real-time semantic reconstruction system.…”

“…Semantic segmentation aims to identify a class label or object category (e.g., chair, table) for every 3D point in a scene while instance segmentation clusters the scene into object instances. These two problems have often been tackled separately in which instance segmentation/detection is a post-processing task of semantic segmentation [31,30]. However, we have observed that object categories and object instances are mutually dependent.…”