RGBD Based Dimensional Decomposition Residual Network for 3D Semantic Scene Completion

Abstract

RGB images differentiate from depth as they carry more details about the color and texture information, which can be utilized as a vital complement to depth for boosting the performance of 3D semantic scene completion (SSC). SSC is composed of 3D shape completion (SC) and semantic scene labeling while most of the existing approaches use depth as the sole input which causes the performance bottleneck. Moreover, the state-of-the-art methods employ 3D CNNs which have cumbersome networks and tremendous parameters. We introduce a light-weight Dimensional Decomposition Residual network (DDR) for 3D dense prediction tasks. The novel factorized convolution layer is effective for reducing the network parameters, and the proposed multi-scale fusion mechanism for depth and color image can improve the completion and segmentation accuracy simultaneously. Our method demonstrates excellent performance on two public datasets. Compared with the latest method SSCNet, we achieve 5.9% gains in SC-IoU and 5.7% gains in SSC-IOU, albeit with only 21% network parameters and 16.6% FLOPs employed compared with that of SSCNet.

Attention-Based Multi-Modal Fusion Network for Semantic Scene Completion

“…We validate our approach on both the synthetic SUNCG-RGBD dataset (Liu et al 2018) and the real NYUv2 dataset (Silberman et al 2012), and the results show that our method achieves a gain of 2.5% on the SUNCG-RGBD dataset and a gain of 2.6% on the NYUv2 dataset against the state-of-the-art method (Li et al 2019b). Our analytical study also observes that directly boosting the AMFNet from the 2D segmentation ground truth of the input RGB-D images (also see Figure 1(f)) can achieve a gain of 3.8% on the synthetic SUNCG-RGBD dataset and a gain of 4.3% on the real NYUv2 dataset against the state-of-the-art method (Li et al 2019b).…”

“…Given a single depth image or RGB-D images of a 3D scene, many papers (Gupta, Arbelaez, and Malik 2013;Ren, Bo, and Fox 2012;Firman et al 2016) have been proposed to complete or segment the 3D scene with neural networks. More recently, a set of methods (Song et al 2017;Liu et al 2018;Guo and Tong 2018;Zhang et al 2018;Li et al 2019b) have been developed to automatically predict the semantic labels, together with completing the 3D geometry, of the objects in a 3D scene from a single view of the scene using convolutional neural networks.…”

“…This line of literature focusing on the task of semantic scene completion (SSC) was initiated by those methods (Song et al 2017;Zhang et al 2018) which take as input only the depth information. Several later works (Liu et al 2018;Li et al 2019b;Guedes, de Campos, and Hilton 2018) argue that the RGB color information, which is often captured together with the depth information by an RGB-D sensor, could be used to improve the performance of the networks which take only the depth information. The experimental results from these works reveal that the RGB color channels provide complementary information to the depth information and higher accuracy of SSC can be achieved by using the fused information.…”

“…Our main contributions can be summarized as follows: • A multi-modal fusion network boosted from 2D semantic segmentation is proposed for semantic scene completion. Compared to previous works (Song et al 2017;Liu et al 2018;Li et al 2019b), the major advantage of this network is that 2D semantic segmentation is used to guide and improve the 3D feature extraction for the SSC task. We demonstrate this advantage can significantly boost the overall performance of the task of SSC by improving both scene completion and segmentation.…”

Depth Based Semantic Scene Completion With Position Importance Aware Loss

“…To enrich the input information and boost the accuracy of SSC, TS3D [12] and DDR-SSC [17] proposed to add a RGB branch in addition to the voxel branch, which introduce extra network or parameters, and are less accurate than our method.…”