Sparsity Invariant CNNs

Uhrig, Jonas; Schneider, N. M.; Schneider, Lukas; Franke, Uwe; Brox, Thomas; Geiger, Andreas

doi:10.1109/3dv.2017.00012

Cited by 781 publications

(934 citation statements)

References 58 publications

Supporting

Mentioning

910

Contrasting

Unclassified

Order By: Relevance

“…To that end, we match our results only to results from publicly available techniques. In terms of raw RMSE error we outperform classical methods such as [4] and [3], as well as the only published CNN method [6]. Qualitatively, our method also better preserves object boundaries which is visible from the results shown on figure 3.…”

Section: Discussionsupporting

confidence: 65%

“…Standard 2D convolution operations have difficulties in learning sparse data input problems [6], [11]. This is especially true when it is necessary to distinguish between actual measurement values and invalid pixels.…”

Section: General Architecturementioning

confidence: 99%

“…The very recently published KITTI depth completion 2 data [6,17,18] is an excellent example of such dataset in the context of depth sensing for autonomous navigation. It consists of video sequences captured by a stereo RGB camera pair as well as point clouds from the Velodyne HDL-64E LiDAR.…”

Section: Generalmentioning

confidence: 99%

“…Uhrig et al [6] propose a depth completion method by processing the raw RGB-D data cube using a novel neural network. They propose a sparsity invariant convolutional layer which is built using an additional sampling mask.…”

Section: Overviewmentioning

confidence: 99%

See 3 more Smart Citations

Learning Morphological Operators for Depth Completion

Dimitrievski

Veelaert

Philips

2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Depth images generated by direct projection of LiDAR point clouds on the image plane suffer from a great level of sparsity which is difficult to interpret by classical computer vision algorithms. We propose a method for completing sparse depth images in a semantically accurate manner by training a novel morphological neural network. Our method approximates morphological operations by Contraharmonic Mean Filter layers which are easily trained in a contemporary deep learning framework. An early fusion U-Net architecture then combines dilated depth channels and RGB using multi-scale processing. Using a large scale RGB-D dataset we are able to learn the optimal morphological and convolutional filter shapes that produce an accurate and fully sampled depth image at the output. Independent experimental evaluation confirms that our method outperforms classical image restoration techniques as well as current state-of-the-art neural networks. The resulting depth images preserve object boundaries and can easily be used to augment various tasks in intelligent vehicles perception systems.

show abstract

Section: Discussionsupporting

confidence: 65%

Section: General Architecturementioning

confidence: 99%

Section: Generalmentioning

confidence: 99%

Section: Overviewmentioning

confidence: 99%

See 2 more Smart Citations

Learning Morphological Operators for Depth Completion

Dimitrievski

Veelaert

Philips

2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Notwithstanding, several recent works have studied learning from sparse labels from different perspectives. A recent work (Uhrig et al, 2017) proposes a new CNN architecture, sparsity invariant CNN, focused on reconstructing a dense depth map from sparse LiDAR information. This approach outputs continuous values in contrast to the classification labels.…”

Section: Models For Weakly Labeled Datamentioning

confidence: 99%

CoralSeg: Learning coral segmentation from sparse annotations

Alonso

Yuval

Eyal

et al. 2019

Journal of Field Robotics

View full text Add to dashboard Cite

Robotic advances and developments in sensors and acquisition systems facilitate the collection of survey data in remote and challenging scenarios. Semantic segmentation, which attempts to provide per‐pixel semantic labels, is an essential task when processing such data. Recent advances in deep learning approaches have boosted this task's performance. Unfortunately, these methods need large amounts of labeled data, which is usually a challenge in many domains. In many environmental monitoring instances, such as the coral reef example studied here, data labeling demands expert knowledge and is costly. Therefore, many data sets often present scarce and sparse image annotations or remain untouched in image libraries. This study proposes and validates an effective approach for learning semantic segmentation models from sparsely labeled data. Based on augmenting sparse annotations with the proposed adaptive superpixel segmentation propagation, we obtain similar results as if training with dense annotations, significantly reducing the labeling effort. We perform an in‐depth analysis of our labeling augmentation method as well as of different neural network architectures and loss functions for semantic segmentation. We demonstrate the effectiveness of our approach on publicly available data sets of different real domains, with the emphasis on underwater scenarios—specifically, coral reef semantic segmentation. We release new labeled data as well as an encoder trained on half a million coral reef images, which is shown to facilitate the generalization to new coral scenarios.

show abstract

FastFusion: Deep stereo‐LiDAR fusion for real‐time high‐precision dense depth sensing

Meng

Li²,

Zhong

et al. 2023

Journal of Field Robotics

View full text Add to dashboard Cite

Light detection and ranging (LiDAR) and stereo cameras are two generally used solutions for perceiving 3D information. The complementary properties of these two sensor modalities motivate a fusion to derive practicable depth sensing toward real‐world applications. Promoted by deep neural network (DNN) techniques, recent works achieve superior performance on accuracy. However, the complex architecture and the sheer number of DNN parameters often lead to poor generalization capacity and non‐real‐time computing. In this paper, we present FastFusion, a three‐stage stereo‐LiDAR deep fusion scheme, which integrates the LiDAR priors into each step of classical stereo‐matching taxonomy, gaining high‐precision dense depth sensing in a real‐time manner. We integrate stereo‐LiDAR information by taking advantage of a compact binary neural network and utilize the proposed cross‐based LiDAR trust aggregation to further fuse the sparse LiDAR measurements in the back‐end of stereo matching. To align the photometrical of the input image and the depth of the estimation, we introduce a refinement network to guarantee consistency. More importantly, we present a graphic processing unit‐based acceleration framework for providing a low‐latency implementation of FastFusion, gaining both accuracy improvement and real‐time responsiveness. In the experiments, we demonstrate the effectiveness and practicability of FastFusion, which obtains a significant speedup over state‐of‐the‐art baselines while achieving comparable accuracy on depth sensing. The video demo for real‐time depth estimation of FastFusion on the real‐world driving scenario is available at https://youtu.be/nP7cls2BA8s.

show abstract

Sparsity Invariant CNNs

Cited by 781 publications

References 58 publications

Learning Morphological Operators for Depth Completion

Learning Morphological Operators for Depth Completion

CoralSeg: Learning coral segmentation from sparse annotations

FastFusion: Deep stereo‐LiDAR fusion for real‐time high‐precision dense depth sensing

Contact Info

Product

Resources

About