Indoor Depth Completion with Boundary Consistency and Self-Attention

Abstract

Depth estimation features are helpful for 3D recognition. Commodity-grade depth cameras are able to capture depth and color image in real-time. However, glossy, transparent or distant surface cannot be scanned properly by the sensor. As a result, enhancement and restoration from sensing depth is an important task. Depth completion aims at filling the holes that sensors fail to detect, which is still a complex task for machine to learn. Traditional hand-tuned methods have reached their limits, while neural network based methods tend to copy and interpolate the output from surrounding depth values. This leads to blurred boundaries, and structures of the depth map are lost. Consequently, our main work is to design an end-to-end network improving completion depth maps while maintaining edge clarity. We utilize self-attention mechanism, previously used in image inpainting fields, to extract more useful information in each layer of convolution so that the complete depth map is enhanced. In addition, we propose boundary consistency concept to enhance the depth map quality and structure. Experimental results validate the effectiveness of our selfattention and boundary consistency schema, which outperforms previous state-of-the-art depth completion work on Matterport3D dataset. Our code is publicly available at

Depth Completion and Super-Resolution with Arbitrary Scale Factors for Indoor Scenes

“…The RMSE and MAE were calculated in meters. The results showed that the proposed method with the Guidance branch based on UNet was slightly worse than the method proposed in [30]. However, it is noticeable that it had a higher percentage of inliers, as shown by the δt and SSIM values.…”

“…The best result is shown in bold. The RMSE of the FCN is from [21], that for the MRF is from [57], and that for the bilateral filter was given in [30]. For the RMSE and MAE, lower is better.…”

“…For example, the network cannot know the shape of an object or where the boundary of an object is. Surface normals and occlusion boundaries estimated by the DCNN from color images were used as guidance information to complete the missing regions in [ 13 , 30 ]. However, as shown in [ 24 , 25 ], the network can also estimate the depth values from just a color image.…”

“…Then, the depth values were interpolated, based on the local neighborhood and the extracted structural information. In [ 30 ], Huang et al applied the contextual attention mechanism to modify the interpolation, based on the depth completion method of [ 13 ]. Although the method achieved high accuracy, using multiple submodules of the DCNN is computationally expensive.…”

“…Although the method achieved high accuracy, using multiple submodules of the DCNN is computationally expensive. Furthermore, References [ 13 , 30 ] only exploited the guidance information for depth completion based on the predicted surface normals and occlusion boundaries estimated by the DCNN. This means that the error in the prediction of the surface normals and the occlusion boundaries can mislead the depth completion process.…”