Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor Data

Lopez-Rodriguez, Adrian; Busam, Benjamin; Mikolajczyk, Krystian

doi:10.1007/978-3-030-69525-5_20

Cited by 21 publications

(11 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Compared to unsupervised methods (Table 10), we rank first amongst all methods with the best scores across all metrics. Our model even beat methods [24,49,44] that use additional synthetic data (Virtual KITTI [12]) for training, amongst which is the state of the art [44]. Despite this, we beat [44] by an average of 8% across all metrics while using 11.5% fewer parameters.…”

Section: E Kitti Depth Completion Benchmarkmentioning

confidence: 79%

“…Unlike [49], our method does not require ground truth and is not limited to a specific domain. [24,44] leverage additional synthetic datasets, which require dealing with simto-real; our method is able to achieve the state-of-the-art without needing access to additional data.…”

Section: Related Work and Contributionsmentioning

confidence: 99%

“…Overall, we beat the best performing method [44] by an average of 8% and up to 10.4% on the iMAE metric with a 11.5% reduction in model size. We note that top methods [24,44] use additional synthetic data Figure 4: Qualitative results on KITTI test set. Head-to-head comparison against [46].…”

Section: Kitti Depth Completion Benchmarkmentioning

confidence: 99%

See 2 more Smart Citations

Unsupervised Depth Completion with Calibrated Backprojection Layers

Wong¹,

Soatto²

2021

Preprint

View full text Add to dashboard Cite

We propose a deep neural network architecture to infer dense depth from an image and a sparse point cloud. It is trained using a video stream and corresponding synchronized sparse point cloud, as obtained from a LIDAR or other range sensor, along with the intrinsic calibration parameters of the camera. At inference time, the calibration of the camera, which can be different than the one used for training, is fed as an input to the network along with the sparse point cloud and a single image. A Calibrated Backprojection Layer backprojects each pixel in the image to three-dimensional space using the calibration matrix and a depth feature descriptor. The resulting 3D positional encoding is concatenated with the image descriptor and the previous layer output to yield the input to the next layer of the encoder. A decoder, exploiting skip-connections, produces a dense depth map. The resulting Calibrated Backprojection Network, or KBNet, is trained without supervision by minimizing the photometric reprojection error. KB-Net imputes missing depth value based on the training set, rather than on generic regularization. We test KBNet on public depth completion benchmarks, where it outperforms the state of the art by 30% indoor and 8% outdoor when the same camera is used for training and testing. When the test camera is different, the improvement reaches 62%. Code available at: https://github.com/alexklwong/ calibrated-backprojection-network.

show abstract

Section: E Kitti Depth Completion Benchmarkmentioning

confidence: 79%

Section: Related Work and Contributionsmentioning

confidence: 99%

See 1 more Smart Citation

Unsupervised Depth Completion with Calibrated Backprojection Layers

Wong¹,

Soatto²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Radar suffers from several sources of noise (clutter), complicating its usage in learning-based approaches. The major origins of clutter are multi-path [23] and "see-through" effects, due to the different viewpoints [27] and the physical sensor properties.…”

Section: Addressing Radar Noisementioning

confidence: 99%

R4Dyn: Exploring Radar for Self-Supervised Monocular Depth Estimation of Dynamic Scenes

Gasperini¹,

Koch²,

Dallabetta³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

While self-supervised monocular depth estimation in driving scenarios has achieved comparable performance to supervised approaches, violations of the static world assumption can still lead to erroneous depth predictions of traffic participants, posing a potential safety issue. In this paper, we present R4Dyn, a novel set of techniques to use cost-efficient radar data on top of a self-supervised depth estimation framework. In particular, we show how radar can be used during training as weak supervision signal, as well as an extra input to enhance the estimation robustness at inference time. Since automotive radars are readily available, this allows to collect training data from a variety of existing vehicles. Moreover, by filtering and expanding the signal to make it compatible with learning-based approaches, we address radar inherent issues, such as noise and sparsity. With R4Dyn we are able to overcome a major limitation of self-supervised depth estimation, i.e. the prediction of traffic participants. We substantially improve the estimation on dynamic objects, such as cars by 37% on the challenging nuScenes dataset, hence demonstrating that radar is a valuable additional sensor for monocular depth estimation in autonomous vehicles. Additionally, we plan on making the code publicly available.

show abstract

“…Modern commercial sensors are typically able to acquire RGB, depth, and infrared video streams, although regulatory and technological constraints continue to make acquisition, storage and access challenging in clinical settings [22]. While we can glean substantial knowledge from the computer vision community on integrating these modalities optimally [16,19], the OR presents particular challenges. Inconsistent lighting conditions, homogeneous color scales, and ubiquitous occlusions can yield unexpected results when adapting state-of-the-art methods from other domains.…”

Section: Introductionmentioning

confidence: 99%

Know your sensORs -- A Modality Study For Surgical Action Classification

Bastian¹,

Czempiel²,

Heiliger³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

The surgical operating room (OR) presents many opportunities for automation and optimization. Videos from various sources in the OR are becoming increasingly available. The medical community seeks to leverage this wealth of data to develop automated methods to advance interventional care, lower costs, and improve overall patient outcomes. Existing datasets from OR room cameras are thus far limited in size or modalities acquired, leaving it unclear which sensor modalities are best suited for tasks such as recognizing surgical action from videos. This study demonstrates that surgical action recognition performance can vary depending on the image modalities used. We perform a methodical analysis on several commonly available sensor modalities, presenting two fusion approaches that improve classification performance. The analyses are carried out on a set of multi-view RGB-D video recordings of 18 laparoscopic procedures.

show abstract

Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor Data

Cited by 21 publications

References 51 publications

Unsupervised Depth Completion with Calibrated Backprojection Layers

Unsupervised Depth Completion with Calibrated Backprojection Layers

R4Dyn: Exploring Radar for Self-Supervised Monocular Depth Estimation of Dynamic Scenes

Know your sensORs -- A Modality Study For Surgical Action Classification

Contact Info

Product

Resources

About