DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network

Gu, Jianzhong; Xiang, Zhiyu; Ye, Yuwen; Wang, Lingxuan

doi:10.1109/lra.2021.3060396

Cited by 44 publications

(16 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Gradient-related Methodsmentioning

confidence: 99%

“…Gradient information has been used in previous depth completion works, such as [45][46][47][48][49][50]. Commonly, there are two ways to introduce gradient information into deep networks: 1) incorporating gradients into the model to guide depth completion [45], and 2) introducing gradients into the loss for constraints [45][46][47][48][49][50]. Specifically, Hwang et al [45] designed a teacher network to learn gradient depth images, which were then used to train their geometrical edge CNN through a Knowledge-Distillation loss function.…”

Section: Gradient-related Methodsmentioning

confidence: 99%

“…Nguyen et al [ 46 ] and Ryu et al [ 47 ] adopted a gradient-related loss to encourage local smoothness of depth predictions. Gu et al [ 48 ] used some pseudo depth maps to rectify input sparse depth, and supervised training by some structural losses including gradient loss. Liu et al [ 49 ] applied a constraint on depth gradient to penalize the disagreement of depth boundaries.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

SPNet: Structure preserving network for depth completion

Luo

Fan

et al. 2023

PLoS ONE

View full text Add to dashboard Cite

Depth completion aims to predict a dense depth map from a sparse one. Benefiting from the powerful ability of convolutional neural networks, recent depth completion methods have achieved remarkable performance. However, it is still a challenging problem to well preserve accurate depth structures, such as tiny structures and object boundaries. To tackle this problem, we propose a structure preserving network (SPNet) in this paper. Firstly, an efficient multi-scale gradient extractor (MSGE) is proposed to extract useful multi-scale gradient images, which contain rich structural information that is helpful in recovering accurate depth. The MSGE is constructed based on the proposed semi-fixed depthwise separable convolution. Meanwhile, we adopt a stable gradient MAE loss (LGMAE) to provide additional depth gradient constrain for better structure reconstruction. Moreover, a multi-level feature fusion module (MFFM) is proposed to adaptively fuse the spatial details from low-level encoder and the semantic information from high-level decoder, which will incorporate more structural details into the depth modality. As demonstrated by experiments on NYUv2 and KITTI datasets, our method outperforms some state-of-the-art methods in terms of both quantitative and quantitative evaluations.

show abstract

Section: Gradient-related Methodsmentioning

confidence: 99%

Section: Gradient-related Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

SPNet: Structure preserving network for depth completion

Luo

Fan

et al. 2023

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…Depth completion aims to predict a dense depth map from a sparse one with the guidance of a color image. Recently, many efficient depth completion methods are proposed [8,9,12,14]. [12] utilizes a two-branch backbone to realize a precise and efficient depth completion network.…”

Section: Related Workmentioning

confidence: 99%

Sparse Fuse Dense: Towards High Quality 3D Detection with Depth Completion

Wu¹,

Peng²,

Xie³

et al. 2022

Preprint

View full text Add to dashboard Cite

Current LiDAR-only 3D detection methods inevitably suffer from the sparsity of point clouds. Many multi-modal methods are proposed to alleviate this issue, while different representations of images and point clouds make it difficult to fuse them, resulting in suboptimal performance. In this paper, we present a novel multi-modal framework SFD (Sparse Fuse Dense), which utilizes pseudo point clouds generated from depth completion to tackle the issues mentioned above. Different from prior works, we propose a new RoI fusion strategy 3D-GAF (3D Grid-wise Attentive Fusion) to make fuller use of information from different types of point clouds. Specifically, 3D-GAF fuses 3D RoI features from the pair of point clouds in a grid-wise attentive way, which is more fine-grained and more precise. In addition, we propose a SynAugment (Synchronized Augmentation) to enable our multi-modal framework to utilize all data augmentation approaches tailored to LiDAR-only methods. Lastly, we customize an effective and efficient feature extractor CPConv (Color Point Convolution) for pseudo point clouds. It can explore 2D image features and 3D geometric features of pseudo point clouds simultaneously. Our method holds the highest entry on the KITTI car 3D object detection leaderboard † , demonstrating the effectiveness of our SFD. Code will be made publicly available.

show abstract

“…RMSE and MAE are defined as follows: [41] 730.08 210.55 2.17 0.94 0.032s ACMNet [42] 732.99 206.80 2.08 0.90 0.080s FCFR-Net [43] 735.81 217.15 2.20 0.98 0.130s GuideNet [44] 736.24 218.83 2.25 0.99 0.140s NLSPN [47] 741.68 199.59 1.99 0.84 0.220s CSPN++ [45] 743.69 209.28 2.07 0.90 0.200s UberATG-FuseNet [48] 752.88 221.19 2.34 1.14 0.090s DenseLiDAR [49] 755.41 214.13 2.25 0.96 0.020s DeepLiDAR [10] 758.38 226.50 2.56 1.15 0.070s DANConv [50] 759.65 213.68 2.17 0.92 0.050s…”

Section: A Experimental Setupmentioning

confidence: 99%

NNNet: New Normal Guided Depth Completion From Sparse LiDAR Data and Single Color Image

Liu

Jung

2022

IEEE Access

View full text Add to dashboard Cite

In this paper, we propose new normal guided depth completion from sparse LiDAR data and single color image, named NNNet. Sparse depth completion often uses normal maps as a constraint for model training. However, direct construction of a normal map from the color image causes a lot of noise in the normal map and reduces the model performance. Thus, we use a new normal map as an intermediate constraint to promote the fusion of multi-modal features. We generate the new normal map from the sparse LiDAR depth data to use it as a constraint for network training. The new normal map is generated by converting the input depth into a grayscale image, constructing a normal map, replacing the Z channel of the normal map with the original depth, and finally adding a mask. Based on the new normal map, we construct an end-to-end network NNNet for sparse depth completion guided by its corresponding color image. NNNet consists of two branches. The one branch generates the new normal map from the depth image and its corresponding color image, while the other branch constructs a dense depth image from the sparse depth and the predicted new normal map. The two branches fully merge the features through skip connection. In loss function, we use L2 loss to ensure that the new normal map plays a restrictive role. Finally, we generate the dense depth image by refining it with a spatial propagation network. Experimental results show that the new normal map provides effective constraints for sparse depth completion. Moreover, NNNet achieves 724.14 in terms of RMSE and outperforms most of the current state-of-the-art methods.

show abstract

DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network

Cited by 44 publications

References 31 publications

SPNet: Structure preserving network for depth completion

SPNet: Structure preserving network for depth completion

Sparse Fuse Dense: Towards High Quality 3D Detection with Depth Completion

NNNet: New Normal Guided Depth Completion From Sparse LiDAR Data and Single Color Image

Contact Info

Product

Resources

About