Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Abstract

Depth completion aims to recover dense depth maps from sparse depth measurements. It is of increasing importance for autonomous driving and draws increasing attention from the vision community. Most of existing methods directly train a network to learn a mapping from sparse depth inputs to dense depth maps, which has difficulties in utilizing the 3D geometric constraints and handling the practical sensor noises. In this paper, to regularize the depth completion and improve the robustness against noise, we propose a unified CNN framework that 1) models the geometric constraints between depth and surface normal in a diffusion module and 2) predicts the confidence of sparse Li-DAR measurements to mitigate the impact of noise. Specifically, our encoder-decoder backbone predicts surface normals, coarse depth and confidence of LiDAR inputs simultaneously, which are subsequently inputted into our diffusion refinement module to obtain the final completion results. Extensive experiments on KITTI depth completion dataset and NYU-Depth-V2 dataset demonstrate that our method achieves state-of-the-art performance. Further ablation study and analysis give more insights into the proposed method and demonstrate the generalization capability and stability of our model. AbstractSorry about this little trick and hope it would work, since I do not have much time to neatten my original source code. depth completion aims to recover dense depth maps from sparse depth measurements. It is of increasing importance for autonomous driving and draws increasing attention from the vision community. Most of existing methods directly train a network to learn a mapping from sparse depth inputs to dense depth maps, which has difficulties in utilizing the 3D geometric constraints and handling the practical sensor noises. In this paper, to regularize the depth completion and improve the robustness against noise, we propose a unified CNN framework that 1) models the geometric constraints between depth and surface normal in a

Deep Architecture With Cross Guidance Between Single Image and Sparse LiDAR Data for Depth Completion

“…First, UberATG-Fusenet [29] was divided because it exploits the 3D point cloud of LiDAR data. DeepLidar [26] and PwP [27] use the surface normal information to determine the representation power for the depth completion task. Then, we divided the model trained with only the official dataset and the model trained with the official dataset and additional data.…”

“…Ma et al [20] developed a deep regression model with an early fusion method for direct mapping from the sparse depth to the dense depth and a self-supervised training framework. The surface normal was introduced by [26], [27] to obtain more accurate three-dimensional (3D) geometric information for the depth completion task. The confidence mask for refining the results from the network was the same as in [19].…”

Uncertainty-Aware CNNs for Depth Completion: Uncertainty from Beginning to End

“…Gansbeke et al [33] proposed the use of confidences to fuse two network streams utilizing sparse depth and RGB images respectively. Similarly, Xu et al [36] predict a confidence mask that is used to mitigate the impact of noisy measurements on different components of their network. However, none of these methods provided any prediction uncertainty measure for the final prediction.…”

“…Modeling uncertainty for this task is crucial due to the inherent noisy and sparse nature of depth sensors, caused by multi-path interference and depth ambiguities [11]. Previous approaches proposed to learn some intermediate confidence masks to mitigate the impact of disturbed measurements inside their networks [28,33,36]. However, none of these approaches has demonstrated the probabilistic validity of the intermediate confidence masks.…”

Revisiting Sparsity Invariant Convolution: A Network for Image Guided Depth Completion

“…Comparing with binary mask, this strategy allows network to adaptive adjust confidence signal and maximize output probability at the same time. Xu et al [18] utilize the constraint between surface normal and depth map to guide the depth estimation process. This constraint is formed by the distances between piece-wise planes and the origin.…”