DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image

Abstract

In this paper, we propose a deep learning architecture that produces accurate dense depth for the outdoor scene from a single color image and a sparse depth. Inspired by the indoor depth completion, our network estimates surface normals as the intermediate representation to produce dense depth, and can be trained end-to-end. With a modified encoder-decoder structure, our network effectively fuses the dense color image and the sparse LiDAR depth. To address outdoor specific challenges, our network predicts a confidence mask to handle mixed LiDAR signals near foreground boundaries due to occlusion, and combines estimates from the color image and surface normals with learned attention maps to improve the depth accuracy especially for distant areas. Extensive experiments demonstrate that our model improves upon the state-of-the-art performance on KITTI depth completion benchmark. Ablation study shows the positive impact of each model components to the final performance, and comprehensive analysis shows that our model generalizes well to the input with higher sparsity or from indoor scenes. * indicates equal contributions.† indicates corresponding author. Color Image Sparse Data from LiDARDeepLiDAR: Our Dense Prediction (colored with input color image)DeepLiDAR: Our Dense Prediction (colored with surface normal)

Accurate Monocular 3D Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving

“…In comparison, our method takes the generated depth as the core feature and transform it into 3D space to explicitly make use of its spatial information. Pseudo-LiDAR [31] also find that data presentation plays an important role in 3D detection task. It pays more attention to verify the universality of point cloud representation and applies the generated points to some different existing 3D detection methods without any modifications.…”

Augmented LiDAR Simulator for Autonomous Driving

“…Currently, CARLA, as an open-source simulation platform 3 , have been widely used for different kinds of simulation purposes (e.g., [10] [28] [20] [12] [17]). Similar with most typical simulation frameworks, both the foreground and background CG models have to be built in advance.…”

Aerial Single-View Depth Completion With Image-Guided Uncertainty Estimation

“…Some large networks, e.g. [31], [32], produce intermediary results with some correlation with a confidence measurement, but these results are not meant to be used as output. In this work, the confidence is interpreted as a classification problem in which the confidence indicates the probability of each pixel to have a correct depth estimation.…”

“…Our loss network is inspired by methods that compute multiple depths using different approaches in the same network and then combine the depth results using a weighted sum guided by the also estimated relative-confidence or attention maps, e.g. [31], [32]. A high confidence area in one of the maps means that the correspondent approach is likely to compute a better depth estimation inside this area than the other approaches of the network.…”

“…In order to achieve this type of confidence, we opted for using normalized convolutions [35], as their formulation is also a weighted sum guided by the confidence, as shown in Equation 1. However, this equation is the sum of neighbouring pixels in the convolution instead of multiple depth maps like in [31], [32].…”