ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans

Abstract

3D scans of indoor environments suffer from sensor occlusions, leaving 3D reconstructions with highly incomplete 3D geometry (left). We propose a novel data-driven approach based on fully-convolutional neural networks that transforms incomplete signed distance functions (SDFs) into complete meshes at unprecedented spatial extents (middle). In addition to scene completion, our approach infers semantic class labels even for previously missing geometry (right). Our approach outperforms existing approaches both in terms of completion and semantic labeling accuracy by a significant margin. AbstractWe introduce ScanComplete, a novel data-driven approach for taking an incomplete 3D scan of a scene as input and predicting a complete 3D model along with per-voxel semantic labels. The key contribution of our method is its ability to handle large scenes with varying spatial extent, managing the cubic growth in data size as scene size increases. To this end, we devise a fully-convolutional generative 3D CNN model whose filter kernels are invariant to the overall scene size. The model can be trained on scene subvolumes but deployed on arbitrarily large scenes at test time. In addition, we propose a coarse-to-fine inference strategy in order to produce high-resolution output while also leveraging large input context sizes. In an extensive series of experiments, we carefully evaluate different model design choices, considering both deterministic and probabilistic models for completion and semantic inference. Our results show that we outperform other methods not only in the size of the environments handled and processing efficiency, but also with regard to completion quality and semantic segmentation performance by a significant margin.

Learned Semantic Multi-Sensor Depth Map Fusion

“…We report true positive (TP) rate of completion, average surface distance to the ground truth, semantic accuracy (SA) and free-space accuracy (FA). The comparison demonstrates that our method performs better than the baselines [6,14].…”

“…Top row: Estimated confidence values from input measurement on ScanNet dataset. Middle and bottom rows: The proposed learned fusion in comparison to datacost averaging in [6], ScanComplete [14] and ground truth.…”

End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans

“…We conduct experiments on the SUNCG dataset [32] to verify the effectiveness of our method. For training and evaluation, we create virtual scans of the synthetic scenes, where we simulate a large-scale indoor 3D reconstruction by using rendered depth frames similar to [14,8] with the distinction that we add noise to the synthetic depth frames in the fusion process. The voxel resolution for the generated SDF grids is at 4.68cm.…”

Efficient Semantic Scene Completion Network with Spatial Group Convolution

“…SSCNet [40] combined these two tasks together and showed that segmentation and completion can benefit from each other. In order to generate high resolution 3D structure, various methods had been explored, such as long short-term memorized [15], coarse-to-fine strategy [5], 3D generative adversarial network [47], and inverse discrete cosine transform [22]. Recently, segmentation and completion are both benefited from these advanced 3D deep learning methods described in section 2.1.…”