Estimating Depth from RGB and Sparse Sensing

Zhao, Chen; Badrinarayanan, Vijay; Drozdov, Gilad; Rabinovich, Andrew

doi:10.1007/978-3-030-01225-0_11

Cited by 105 publications

(65 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A line is fitted based on the data. The left comes from subject 05, the middle from subject 18, the right from subject 64. reconstruction from sparse observations [9,24,21,22,7]. These two solutions make our central pipeline of DNN more easily to adapt to handling missing data.…”

Section: Robustness Analysismentioning

confidence: 99%

Deep Non-Rigid Structure From Motion

Kong

Lucey

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Non-Rigid Structure from Motion (NRSfM) refers to the problem of reconstructing cameras and the 3D point cloud of a non-rigid object from an ensemble of images with 2D correspondences. Current NRSfM algorithms are limited from two perspectives: (i) the number of images, and (ii) the type of shape variability they can handle. These difficulties stem from the inherent conflict between the condition of the system and the degrees of freedom needing to be modeled -which has hampered its practical utility for many applications within vision. In this paper we propose a novel hierarchical sparse coding model for NRSFM which can overcome (i) and (ii) to such an extent, that NRSFM can be applied to problems in vision previously thought too ill posed. Our approach is realized in practice as the training of an unsupervised deep neural network (DNN) auto-encoder with a unique architecture that is able to disentangle pose from 3D structure. Using modern deep learning computational platforms allows us to solve NRSfM problems at an unprecedented scale and shape complexity. Our approach has no 3D supervision, relying solely on 2D point correspondences. Further, our approach is also able to handle missing/occluded 2D points without the need for matrix completion. Extensive experiments demonstrate the impressive performance of our approach where we exhibit superior precision and robustness against all available state-of-the-art works in some instances by an order of magnitude. We further propose a new quality measure (based on the network weights) which circumvents the need for 3D ground-truth to ascertain the confidence we have in the reconstructability. We believe our work to be a significant advance over state-of-the-art in NRSFM.Index Terms-Nonrigid structure from motion, hierarchical sparse coding, deep neural network, reconstructability, missing data. ! • C. Kong and S. Lucey are with the

show abstract

Section: Robustness Analysismentioning

confidence: 99%

Deep Non-Rigid Structure From Motion

Kong

Lucey

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

show abstract

“…Huang et al propose the use of additional sparse operations such as sparsity invariant upsampling, addition and concatenation in addition to convolution and were able to achieve much better results [27]. However, we are more inclined to believe that desirable performance can be achieved with the use of regular convolutions and operations for multi-modal input with simple pre-processing hole filling operations such as morphological filters, fill maps and nearest neighbor interpolation [28], [14], [44]. This is simple and effective in providing the network with a good initialization.…”

Section: A Architecturementioning

confidence: 99%

DFuseNet: Deep Fusion of RGB and Sparse Depth Information for Image Guided Dense Depth Completion

Shivakumar

Nguyen

Miller

et al. 2019

2019 IEEE Intelligent Transportation Systems Conference (ITSC)

100

View full text Add to dashboard Cite

In this paper we propose a convolutional neural network that is designed to upsample a series of sparse range measurements based on the contextual cues gleaned from a high resolution intensity image. Our approach draws inspiration from related work on super-resolution and in-painting. We propose a novel architecture that seeks to pull contextual cues separately from the intensity image and the depth features and then fuse them later in the network. We argue that this approach effectively exploits the relationship between the two modalities and produces accurate results while respecting salient image structures. We present experimental results to demonstrate that our approach is comparable with state of the art methods and generalizes well across multiple datasets.1

show abstract

“…Loss function for depth completion A key component of depth completion is the choice of loss function. Recent work has explored loss functions including L2 [4,19], L1 [20], inverse-L1 [15], and softmax losses on depth [16]. While these loss functions can achieve low error on measures including RMSE, MAE, iMAE, often it comes at the cost of smoothing out depth estimate at object boundaries.…”

Section: Related Workmentioning

confidence: 99%

Depth Coefficients for Depth Completion

Imran

Long

Liu

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Depth completion involves estimating a dense depth image from sparse depth measurements, often guided by a color image. While linear upsampling is straight forward, it results in artifacts including depth pixels being interpolated in empty space across discontinuities between objects. Current methods use deep networks to upsample and "complete" the missing depth pixels. Nevertheless, depth smearing between objects remains a challenge. We propose a new representation for depth called Depth Coefficients (DC) to address this problem. It enables convolutions to more easily avoid inter-object depth mixing. We also show that the standard Mean Squared Error (MSE) loss function can promote depth mixing, and thus propose instead to use cross-entropy loss for DC. With quantitative and qualitative evaluation on benchmarks, we show that switching out sparse depth input and MSE loss with our DC representation and crossentropy loss is a simple way to improve depth completion performance, and reduce pixel depth mixing, which leads to improved depth-based object detection.

show abstract

Estimating Depth from RGB and Sparse Sensing

Cited by 105 publications

References 37 publications

Deep Non-Rigid Structure From Motion

Deep Non-Rigid Structure From Motion

DFuseNet: Deep Fusion of RGB and Sparse Depth Information for Image Guided Dense Depth Completion

Depth Coefficients for Depth Completion

Contact Info

Product

Resources

About