View-Volume Network for Semantic Scene Completion from a Single Depth Image

Abstract

We introduce a View-Volume convolutional neural network (VVNet) for inferring the occupancy and semantic labels of a volumetric 3D scene from a single depth image. The VVNet concatenates a 2D view CNN and a 3D volume CNN with a differentiable projection layer. Given a single RGBD image, our method extracts the detailed geometric features from the input depth image with a 2D view CNN and then projects the features into a 3D volume according to the input depth map via a projection layer. After that, we learn the 3D context information of the scene with a 3D volume CNN for computing the result volumetric occupancy and semantic labels. With combined 2D and 3D representations, the VVNet efficiently reduces the computational cost, enables feature extraction from multi-channel high resolution inputs, and thus significantly improves the result accuracy. We validate our method and demonstrate its efficiency and effectiveness on both synthetic SUNCG and real NYU dataset.

ForkNet: Multi-Branch Volumetric Semantic Completion From a Single Depth Image

“…Focusing on the semantic segmentation on the observed surface, our approach performs at an IoU of 57.2% which is 3.0% higher than SSCNet [30]. On the other hand, when we evaluate the IoU measure on the entire volume in Table 1, our method reaches an average IoU of 63.4% which is significantly better than Wang et al [34], 3D-RecGAN [38] and SSCNet [30] but slightly worse than VVNet [12] and SaTNet [21].…”

“…floor wall win. chair bed sofa [20], 3D-RecGAN [38], Geiger and Wang [9], SSCNet [30], VVNet [12], and SaTNet [21]. The resolution of our input volume is given in the scale of 80×48×80 voxels.…”

“…The resolution of our input volume is given in the scale of 80×48×80 voxels. While [9,12,20,21,30] produce 60×36×60 semantic volumes for evaluation, [34,38] and us produce a slightly higher resolution of 80×48×80.…”

Depth Based Semantic Scene Completion With Position Importance Aware Loss

“…Recently several methods have been proposed for SSC using deep learning techniques [3], [12], [13], [8]. Among them, the most representative work is the SSCNet [3] which conducts the semantic labeling and scene completion simultaneously and also proves that these two tasks can benefit from each other.…”

“…3D convolutional neural network (3D-CNN) is frequently used in the task of 3D scene prediction. Since 3D-CNN requires a regular grid as input, voxels are naturally chosen to represent the 3D scene in these 3D-CNN based methods, and different loss functions are employed for training them [3], [8], [9].…”