Active Scene Understanding via Online Semantic Reconstruction

Abstract

We propose a novel approach to robot‐operated active understanding of unknown indoor scenes, based on online RGBD reconstruction with semantic segmentation. In our method, the exploratory robot scanning is both driven by and targeting at the recognition and segmentation of semantic objects from the scene. Our algorithm is built on top of a volumetric depth fusion framework and performs real‐time voxel‐based semantic labeling over the online reconstructed volume. The robot is guided by an online estimated discrete viewing score field (VSF) parameterized over the 3D space of 2D location and azimuth rotation. VSF stores for each grid the score of the corresponding view, which measures how much it reduces the uncertainty (entropy) of both geometric reconstruction and semantic labeling. Based on VSF, we select the next best views (NBV) as the target for each time step. We then jointly optimize the traverse path and camera trajectory between two adjacent NBVs, through maximizing the integral viewing score (information gain) along path and trajectory. Through extensive evaluation, we show that our method achieves efficient and accurate online scene parsing during exploratory scanning.

State‐of‐the‐art in Automatic 3D Reconstruction of Structured Indoor Environments

“…With few exceptions, however, RGB and 3D data are generally analyzed separately in the reconstruction process, most of the time exploiting RGB analysis for 3D data densification prior to the application of a pure geometric processing pipeline. Performing data fusion to combine visual and depth cues into multi‐modal feature descriptors on which to base further analysis is an important avenue for future work: such a joint analysis allows to better cope with heavily cluttered and partial acquisitions, as demonstrated by early results on boundary surface reconstruction [LWF18a] and indoor object reconstruction [SFCH12, ZZZ∗19, JDN19].…”

“…While the above approaches perform object recognition on images or inside the reconstruction step, 3D object segmentation can also be performed over the 3D reconstruction of scene geometry, in order to facilitate 3D spatial and structural reasoning [ZXTZ15, XHS∗15], at least when a dense input is available. In this context, Hou et al [JDN19] and Zheng et al [ZZZ∗19] have recently proposed methods for active scene understanding based on online RGB‐D reconstruction with volumetric segmentation. In those approaches, a deep neural network is leveraged to perform real‐time voxel‐based semantic labeling.…”

InstanceFusion: Real‐time Instance‐level 3D Reconstruction Using a Single RGBD Camera

“…However, all of the above systems suffer from incorrect single‐frame segmentation and error accumulation, resulting in noisy and inconsistent reconstructed objects. Zheng et al [ZZZ*19] achieve semantic understanding of indoor scenes, based on online RGBD reconstruction with volumetric semantic segmentation. However, it is difficult to distinguish different instances.…”

Adding Uncertainty to an Object Detection System for Mobile Robots

“…It is considered a complex task due to the multiple sub-tasks that are involved, such as object recognition, scene classification, geometric reasoning, semantic segmentation, pose estimation, 3D reconstruction, saliency detection, physics-based reasoning, and affordance prediction. Scene understanding task has been addressed in different ways: through the parsing of single images [7,20,89,91,107,176,195,201,205], and considering the understanding of the whole environment where a robot moves [84,147,200]. In the literature, depending on the final goal pursued, the approaches emphasize some sub-tasks more than others.…”

“…. 18 2.6 Results of applying the approach of [200] shows the semantic maps generated considering only scene information. In column (d) are the semantic maps built through only object information.…”