Dense 3D Object Reconstruction from a Single Depth View

Abstract

In this paper, we propose a novel approach, 3D-RecGAN++, which reconstructs the complete 3D structure of a given object from a single arbitrary depth view using generative adversarial networks. Unlike existing work which typically requires multiple views of the same object or class labels to recover the full 3D geometry, the proposed 3D-RecGAN++ only takes the voxel grid representation of a depth view of the object as input, and is able to generate the complete 3D occupancy grid with a high resolution of $256^3$ by recovering the occluded/missing regions. The key idea is to combine the generative capabilities of 3D encoder-decoder and the conditional adversarial networks framework, to infer accurate and fine-grained 3D structures of objects in high-dimensional voxel space. Extensive experiments on large synthetic datasets and real-world Kinect datasets show that the proposed 3D-RecGAN++ significantly outperforms the state of the art in single view 3D object reconstruction, and is able to reconstruct unseen types of objects.

Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era

“…Another way to improve the resolution of volumetric techniques is by using multi-staged approaches [25], [27], [34], [41], [42]. The first stage recovers a low resolution voxel grid, say 32 3 , using an encoder-decoder architecture.…”

“…The subsequent stages, which function as upsampling networks, refine the reconstruction by focusing on local regions. Yang et al [42] used an up-sampling module which simply consists of two up-convolutional layers. This simple up-sampling module upgrades the output 3D shape to a higher resolution of 256 3 .…”

“…In the case of 3D reconstruction, the encoder can be a ConvNet/ResNet [42], [99] or a variational autoencoder (VAE) [17]. The generator decodes the latent vector x into a 3D shape X = g(x).…”

“…Central to GAN is the adversarial loss function used to jointly train the discriminator and the generator. Here x = h(I) where I is the 2D images(s) of the training shape X. Yang et al [42], [99] observed that the original GAN loss function presents an overall loss for both real and fake input. They then proposed to use the WGAN-GP loss [100], [101], which separately represents the loss for generating fake reconstruction pairs and the loss for discriminating fake and real construction pairs, see [100], [101] for the details.…”

Efficient Semantic Scene Completion Network with Spatial Group Convolution

“…SSCNet [40] combined these two tasks together and showed that segmentation and completion can benefit from each other. In order to generate high resolution 3D structure, various methods had been explored, such as long short-term memorized [15], coarse-to-fine strategy [5], 3D generative adversarial network [47], and inverse discrete cosine transform [22]. Recently, segmentation and completion are both benefited from these advanced 3D deep learning methods described in section 2.1.…”