Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation

Jaritz, Maximilian; Charette, Raoul de; Wirbel, Émilie; Perrotton, Xavier; Nashashibi, Fawzi

doi:10.1109/3dv.2018.00017

Cited by 271 publications

(196 citation statements)

References 25 publications

Supporting

Mentioning

194

Contrasting

Order By: Relevance

“…Ma et al concatenated the sparse depth and color image as the inputs of an off-the-shelf network [26] and further explored the feasibility of self-supervised Li-DAR completion [23]. Moreover, [14,16,33,4] proposed different network architectures to better exploit the potential of the encoder-decoder framework. However, the encoderdecoder architecture tends to predict the depth maps comprehensively but fails to concentrate on the local areas.…”

Section: Related Workmentioning

confidence: 99%

“…With the advances of deep learning methods, many depth completion approaches based on convolutional neural networks (CNNs) have been proposed. The mainstream of these methods is to directly input the sparse depth maps (with/without color images) into an encoder-decoder network and predict dense depth maps [26,16,36,15,10,23,2]. These black-box methods force the CNN to learn a mapping from sparse depth measurements to dense maps, which is generally a challenging task and leads to unsatisfactory completion results, as shown in Fig.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Zhu

Shi

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

236

142

View full text Add to dashboard Cite

Depth completion aims to recover dense depth maps from sparse depth measurements. It is of increasing importance for autonomous driving and draws increasing attention from the vision community. Most of existing methods directly train a network to learn a mapping from sparse depth inputs to dense depth maps, which has difficulties in utilizing the 3D geometric constraints and handling the practical sensor noises. In this paper, to regularize the depth completion and improve the robustness against noise, we propose a unified CNN framework that 1) models the geometric constraints between depth and surface normal in a diffusion module and 2) predicts the confidence of sparse Li-DAR measurements to mitigate the impact of noise. Specifically, our encoder-decoder backbone predicts surface normals, coarse depth and confidence of LiDAR inputs simultaneously, which are subsequently inputted into our diffusion refinement module to obtain the final completion results. Extensive experiments on KITTI depth completion dataset and NYU-Depth-V2 dataset demonstrate that our method achieves state-of-the-art performance. Further ablation study and analysis give more insights into the proposed method and demonstrate the generalization capability and stability of our model. AbstractSorry about this little trick and hope it would work, since I do not have much time to neatten my original source code. depth completion aims to recover dense depth maps from sparse depth measurements. It is of increasing importance for autonomous driving and draws increasing attention from the vision community. Most of existing methods directly train a network to learn a mapping from sparse depth inputs to dense depth maps, which has difficulties in utilizing the 3D geometric constraints and handling the practical sensor noises. In this paper, to regularize the depth completion and improve the robustness against noise, we propose a unified CNN framework that 1) models the geometric constraints between depth and surface normal in a

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Zhu

Shi

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

236

142

View full text Add to dashboard Cite

show abstract

“…Also, Ma [22] proposed a self-supervised framework without the need for dense labels, achieving great performance on KITTI [8] dataset. Some works combined semantic segmentation [16] to improve the prediction.…”

Section: Depth Reconstruction From Sparse Samplesmentioning

confidence: 99%

Indoor Depth Completion with Boundary Consistency and Self-Attention

Huang

Liu

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

Depth estimation features are helpful for 3D recognition. Commodity-grade depth cameras are able to capture depth and color image in real-time. However, glossy, transparent or distant surface cannot be scanned properly by the sensor. As a result, enhancement and restoration from sensing depth is an important task. Depth completion aims at filling the holes that sensors fail to detect, which is still a complex task for machine to learn. Traditional hand-tuned methods have reached their limits, while neural network based methods tend to copy and interpolate the output from surrounding depth values. This leads to blurred boundaries, and structures of the depth map are lost. Consequently, our main work is to design an end-to-end network improving completion depth maps while maintaining edge clarity. We utilize self-attention mechanism, previously used in image inpainting fields, to extract more useful information in each layer of convolution so that the complete depth map is enhanced. In addition, we propose boundary consistency concept to enhance the depth map quality and structure. Experimental results validate the effectiveness of our selfattention and boundary consistency schema, which outperforms previous state-of-the-art depth completion work on Matterport3D dataset. Our code is publicly available at

show abstract

“…Triggered by the KITTI depth completion benchmark the following depth augmentation methods have been proposed: [14] explicitly tackle the problem of sparsity in depth maps originating from different depth sensing modalities and employ a convolutional encoder-decoder architecture to predict depth or semantic labels. [15] propose another unsupervised learning method to overcome the lack of suitable training data by employing an unsupervised adversarial learning framework for depth augmentation.…”

Section: A Inference Of 3d Structurementioning

confidence: 99%

Predicting Unobserved Space for Planning via Depth Map Augmentation

Taubner

Liu

Siegwart

et al. 2019

2019 19th International Conference on Advanced Robotics (ICAR)

View full text Add to dashboard Cite

Safe and efficient path planning is crucial for autonomous mobile robots. A prerequisite for path planning is to have a comprehensive understanding of the 3D structure of the robot's environment. On Micro Air Vehicles (MAVs) this is commonly achieved using low-cost sensors, such as stereo or RGB-D cameras. These sensors may fail to provide depth measurements in textureless or IR-absorbing areas and have limited effective range. In path planning, this results in inefficient trajectories or failure to recognize a feasible path to the goal, hence significantly impairing the robot's mobility. Recent advances in deep learning enables us to exploit prior experience about the shape of the world and hence to infer complete depth maps from color images and additional sparse depth measurements. In this work, we present an augmented planning system and investigate the effects of employing stateof-the-art depth completion techniques, specifically trained to augment sparse depth maps originating from RGB-D sensors, semi-dense methods and stereo matchers. We extensively evaluate our approach in online path planning experiments based on simulated data, as well as global path planning experiments based on real world Micro Air Vehicle (MAV) data. We show that our augmented system, provided with only sparse depth perception, can reach on-par performance to ground truth depth input in simulated online planning experiments. On real world MAV data the augmented system demonstrates superior performance compared to a planner based on very dense RGB-D depth maps.

show abstract

Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation

Cited by 271 publications

References 25 publications

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Indoor Depth Completion with Boundary Consistency and Self-Attention

Predicting Unobserved Space for Planning via Depth Map Augmentation

Contact Info

Product

Resources

About