Deep Reinforcement Learning of Volume-Guided Progressive View Inpainting for 3D Point Scene Completion From a Single Depth Image

Abstract

We present a deep reinforcement learning method of progressive view inpainting for 3D point scene completion under volume guidance, achieving high-quality scene reconstruction from only a single depth image with severe occlusion. Our approach is end-to-end, consisting of three modules: 3D scene volume reconstruction, 2D depth map inpainting, and multi-view selection for completion. Given a single depth image, our method first goes through the 3D volume branch to obtain a volumetric scene reconstruction as a guide to the next view inpainting step, which attempts to make up the missing information; the third step involves projecting the volume under the same view of the input, concatenating them to complete the current view depth, and integrating all depth into the point cloud. Since the occluded areas are unavailable, we resort to a deep Q-Network to glance around and pick the next best view for large hole completion progressively until a scene is adequately reconstructed while guaranteeing validity. All steps are learned jointly to achieve robust and consistent results. We perform qualitative and quantitative evaluations with extensive experiments on the SUNCG data, obtaining better results than the state of the art.

Point Cloud Scene Completion of Obstructed Building Facades with Generative Adversarial Inpainting

“…Another approach uses a 3D encoder–predictor convoluted neural network that combines a 3D deep learning architecture with a 3D shape synthesis technique [ 51 ]. Whereas, other approaches complete scenes by inferring points from depth maps [ 12 , 53 ]. ScanComplete [ 8 ]—a large-scale scene completion method from 3D scans—uses fully convoluted neural networks that can be trained on smaller sets and applied to larger scenes, which allows for efficient processing.…”

“…However, to apply these methods, additional segmentation is required to extract individual objects from a larger scene. Recently, methods using deep learning have been developed [8,9,12,[49][50][51][52][53], and some extend beyond single objects to perform semantic scene segmentation [8,9,12,53]. One approach focused on point cloud upsampling [50], creating denser and more uniform sets of points from a sparse input.…”

A Physics-Aware Neural Network Approach for Flow Data Reconstruction From Satellite Observations

“…Song et al (2017) used an end-to-end network SSCNet for scene completion and Guo and Tong (2018) a view-volume CNN that extracts geometric features from 2D depth images. Zhang and Funkhouser (2018) presented an end-to-end architecture for depth inpainting, and Han et al (2019) used multi-view depth completion to predict point cloud representations. A 3D recurrent network has been used to integrate information from only a few input views (Choy et al, 2016), and Xu et al (2016) used spatial and temporal structure of sequential observations to predict a view sequence.…”

Attention-Based Multi-Modal Fusion Network for Semantic Scene Completion

“…Understanding and reconstructing a 3D scene from partial observations is a very important technique which has received increasing research attention in recent years due to its commercial potential in a large variety of robotics and vision tasks such as robotic navigation (Gupta, Arbelaez, and Malik 2013), autonomous driving (Laugier et al 2011), and scene reconstruction (Hays and Efros 2007;Han et al 2019). Given a single depth image or RGB-D images of a 3D scene, many papers (Gupta, Arbelaez, and Malik 2013;Ren, Bo, and Fox 2012;Firman et al 2016) have been proposed to complete or segment the 3D scene with neural networks.…”