Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera

Ma, Fangchang; Cavalheiro, Guilherme Venturelli; Karaman, Sertaç

doi:10.48550/arxiv.1807.00275

Cited by 21 publications

(102 citation statements)

References 4 publications

Supporting

Mentioning

101

Contrasting

Order By: Relevance

“…However, the assistance from other modalities, e.g., color images, can significantly improve the completion accuracy. Ma et al concatenated the sparse depth and color image as the inputs of an off-the-shelf network [26] and further explored the feasibility of self-supervised Li-DAR completion [23]. Moreover, [14,16,33,4] proposed different network architectures to better exploit the potential of the encoder-decoder framework.…”

Section: Related Workmentioning

confidence: 99%

“…With the advances of deep learning methods, many depth completion approaches based on convolutional neural networks (CNNs) have been proposed. The mainstream of these methods is to directly input the sparse depth maps (with/without color images) into an encoder-decoder network and predict dense depth maps [26,16,36,15,10,23,2]. These black-box methods force the CNN to learn a mapping from sparse depth measurements to dense maps, which is generally a challenging task and leads to unsatisfactory completion results, as shown in Fig.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Zhu

Shi

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

236

142

View full text Add to dashboard Cite

Depth completion aims to recover dense depth maps from sparse depth measurements. It is of increasing importance for autonomous driving and draws increasing attention from the vision community. Most of existing methods directly train a network to learn a mapping from sparse depth inputs to dense depth maps, which has difficulties in utilizing the 3D geometric constraints and handling the practical sensor noises. In this paper, to regularize the depth completion and improve the robustness against noise, we propose a unified CNN framework that 1) models the geometric constraints between depth and surface normal in a diffusion module and 2) predicts the confidence of sparse Li-DAR measurements to mitigate the impact of noise. Specifically, our encoder-decoder backbone predicts surface normals, coarse depth and confidence of LiDAR inputs simultaneously, which are subsequently inputted into our diffusion refinement module to obtain the final completion results. Extensive experiments on KITTI depth completion dataset and NYU-Depth-V2 dataset demonstrate that our method achieves state-of-the-art performance. Further ablation study and analysis give more insights into the proposed method and demonstrate the generalization capability and stability of our model. AbstractSorry about this little trick and hope it would work, since I do not have much time to neatten my original source code. depth completion aims to recover dense depth maps from sparse depth measurements. It is of increasing importance for autonomous driving and draws increasing attention from the vision community. Most of existing methods directly train a network to learn a mapping from sparse depth inputs to dense depth maps, which has difficulties in utilizing the 3D geometric constraints and handling the practical sensor noises. In this paper, to regularize the depth completion and improve the robustness against noise, we propose a unified CNN framework that 1) models the geometric constraints between depth and surface normal in a

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Zhu

Shi

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

236

142

View full text Add to dashboard Cite

show abstract

“…The depth estimation problem can be further divided into different categories according to different sensor inputs. It includes depth in-painting for RGB-D camera and depth completion for planar LIDAR [3], [4].…”

Section: Related Workmentioning

confidence: 99%

“…However, this method is insufficient for the perception system of autonomous vehicles, as multimodal sensing data need to be fused together. To better fuse the input from the RGB image and sparse LIDAR, [3], [9] proposed a self-supervised training pipeline that takes the sequential information of the RGB image as a geometry constraint of the optimization. However, the performance of this architecture is highly reliant on the accuracy of the transition relation between nearby frames, which can be influenced by moving objects within the scene.…”

Section: Related Workmentioning

confidence: 99%

Depth Completion via Inductive Fusion of Planar LIDAR and Monocular Camera

Dong

Mertz

et al. 2020

2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Modern high-definition LIDAR is expensive for commercial autonomous driving vehicles and small indoor robots. An affordable solution to this problem is fusion of planar LIDAR with RGB images to provide a similar level of perception capability. Even though state-of-the-art methods provide approaches to predict depth information from limited sensor input, they are usually a simple concatenation of sparse LIDAR features and dense RGB features through an end-toend fusion architecture. In this paper, we introduce an inductive late-fusion block which better fuses different sensor modalities inspired by a probability model. The proposed demonstration and aggregation network propagates the mixed context and depth features to the prediction network and serves as a prior knowledge of the depth completion. This late-fusion block uses the dense context features to guide the depth prediction based on demonstrations by sparse depth features. In addition to evaluating the proposed method on benchmark depth completion datasets including NYUDepthV2 and KITTI, we also test the proposed method on a simulated planar LIDAR dataset. Our method shows promising results compared to previous approaches on both the benchmark datasets and simulated dataset with various 3D densities.

show abstract

“…Teed and Deng [48] proposed an iterative method to regress dense correspondences from pairs of depth frames and compute the 6-DoF estimate using a PnP [29] algorithm. More recently, the authors of [35] use a model-based pose estimation solution via Perspectiven-Point to recover 6 DoF pose estimates from monocular videos and use the estimate as a form of supervision to enable semi-supervised depth learning from unlabeled videos and LiDAR. Our work borrows a similar concept, however, we take advantage of the model-based PnP solution and the inliers established to outfit a fully differentiable pose estimation module within the 3D keypoint learning frame- We illustrate the overall architecture of our proposed method that uses two consecutive images (target It and source Is) as input to selfsupervise 3D keypoint learning for monocular ego-motion estimation.…”

Section: Related Workmentioning

confidence: 99%

Self-Supervised 3D Keypoint Learning for Ego-motion Estimation

Tang¹,

Ambruş²,

Guizilini³

et al. 2019

Preprint

View full text Add to dashboard Cite

Generating reliable illumination and viewpoint invariant keypoints is critical for feature-based SLAM and SfM. State-of-the-art learning-based methods often rely on generating training samples by employing homography adaptation to create 2D synthetic views. While such approaches trivially solve data association between views, they cannot effectively learn from real illumination and non-planar 3D scenes. In this work, we propose a fully self-supervised approach towards learning depth-aware keypoints purely from unlabeled videos by incorporating a differentiable pose estimation module that jointly optimizes the keypoints and their depths in a Structure-from-Motion setting. We introduce 3D Multi-View Adaptation, a technique that exploits the temporal context in videos to self-supervise keypoint detection and matching in an end-to-end differentiable manner. Finally, we show how a fully self-supervised keypoint detection and description network can be trivially incorporated as a frontend into a state-of-the-art visual odometry framework that is robust and accurate. † * Equal contribution. This work was part of an internship stay at TRI.

show abstract

Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera

Cited by 21 publications

References 4 publications

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Depth Completion via Inductive Fusion of Planar LIDAR and Monocular Camera

Self-Supervised 3D Keypoint Learning for Ego-motion Estimation

Contact Info

Product

Resources

About