An Adaptive Framework for Learning Unsupervised Depth Completion

Wong, Alex; Fei, Xiaohan; Hong, Byung-Woo; Soatto, Stefano

doi:10.1109/lra.2021.3062602

Cited by 29 publications

(8 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Both stereo [26], [36] and monocular [21], [31], [32], [33] paradigms learn dense depth from an image and sparse depth measurements by minimizing the photometric error between the input image and its reconstruction from other views along with the difference between prediction and sparse depth input (sparse depth reconstruction). [21] used Perspective-n-Point [19] and RANSAC [9] to align consecutive frames and [34], [32] proposed an adaptive weighting framework. [36] also used synthetic data but require image, sparse and ground-truth depth.…”

Section: Related Workmentioning

confidence: 99%

Learning Topology From Synthetic Data for Unsupervised Depth Completion

Wong

Cicek

Soatto

2021

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

We present a method for inferring dense depth maps from images and sparse depth measurements by leveraging synthetic data to learn the association of sparse point clouds with dense natural shapes, and using the image as evidence to validate the predicted depth map. Our learned prior for natural shapes uses only sparse depth as input, not images, so the method is not affected by the covariate shift when attempting to transfer learned models from synthetic data to real ones. This allows us to use abundant synthetic data with ground truth to learn the most difficult component of the reconstruction process, which is topology estimation, and use the image to refine the prediction based on photometric evidence. Our approach uses fewer parameters than previous methods, yet, achieves the state of the art on both indoor and outdoor benchmark datasets. Code available at: https://github.com/alexklwong/learning-topology-synthetic-data

show abstract

Section: Related Workmentioning

confidence: 99%

Learning Topology From Synthetic Data for Unsupervised Depth Completion

Wong

Cicek

Soatto

2021

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Selfsupervised or unsupervised depth completion assumes that the ground-truth dense depth is not available. To address this task, a main solution is minimizing the depth loss between the predicted depth and sparse map, and the photometric loss between the RBG image and its warped image [11], [12], [29]- [31]. For example, Ma et al [29] utilized Perspective-n-Point (PnP) [32] and Random Sample Consensus (RANSAC) [33] to estimate the relative pose between two adjacent frames and warp image.…”

Section: Related Workmentioning

confidence: 99%

MetaComp: Learning to Adapt for Online Depth Completion

Chen¹,

Song²,

Ji³

et al. 2022

Preprint

View full text Add to dashboard Cite

Relying on deep supervised or self-supervised learning, previous methods for depth completion from paired single image and sparse depth data have achieved impressive performance in recent years. However, facing a new environment where the test data occurs online and differs from the training data in the RGB image content and depth sparsity, the trained model might suffer severe performance drop. To encourage the trained model to work well in such conditions, we expect it to be capable of adapting to the new environment continuously and effectively.To achieve this, we propose MetaComp. It utilizes the metalearning technique to simulate adaptation policies during the training phase, and then adapts the model to new environments in a self-supervised manner in testing. Considering that the input is multi-modal data, it would be challenging to adapt a model to variations in two modalities simultaneously, due to significant differences in structure and form of the two modal data. Therefore, we further propose to disentangle the adaptation procedure in the basic meta-learning training into two steps, the first one focusing on the depth sparsity while the second attending to the image content. During testing, we take the same strategy to adapt the model online to new multi-modal data. Experimental results and comprehensive ablations show that our MetaComp is capable of adapting to the depth completion in a new environment effectively and robust to changes in different modalities.

show abstract

“…Unsupervised/Self-supervised depth completion assumes stereo images or monocular videos to be available during training. Both stereo [38,49] and monocular [25,44,45,46] training paradigms leverage sparse depth reconstruction and photometric reprojection error as a training signal by minimizing photometric discrepancies between the input image and its reconstruction from other views. [25] used Perspective-n-Point [22] and RANSAC [11] to align consecutive video frames.…”

Section: Related Work and Contributionsmentioning

confidence: 99%

“…Motivation. Unsupervised methods [25,44,45,46] use the photometric reprojection error perl as a training signal. The input image I t is reconstructed from temporally adjacent frames…”

Section: Kbnet Architecturementioning

confidence: 99%

“…We note that the photometric term can be replaced with more sophisticated measures of reprojection error [13] and additional regularizers such as pose consistency [46] or adaptive regularization weighting schemes [47,45] -which would likely boost performance even more. However, we choose a simple loss to demonstrate the efficacy of our novel architecture.…”

Section: A1 Loss Functionmentioning

confidence: 99%

See 1 more Smart Citation

Unsupervised Depth Completion with Calibrated Backprojection Layers

Wong¹,

Soatto²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We propose a deep neural network architecture to infer dense depth from an image and a sparse point cloud. It is trained using a video stream and corresponding synchronized sparse point cloud, as obtained from a LIDAR or other range sensor, along with the intrinsic calibration parameters of the camera. At inference time, the calibration of the camera, which can be different than the one used for training, is fed as an input to the network along with the sparse point cloud and a single image. A Calibrated Backprojection Layer backprojects each pixel in the image to three-dimensional space using the calibration matrix and a depth feature descriptor. The resulting 3D positional encoding is concatenated with the image descriptor and the previous layer output to yield the input to the next layer of the encoder. A decoder, exploiting skip-connections, produces a dense depth map. The resulting Calibrated Backprojection Network, or KBNet, is trained without supervision by minimizing the photometric reprojection error. KB-Net imputes missing depth value based on the training set, rather than on generic regularization. We test KBNet on public depth completion benchmarks, where it outperforms the state of the art by 30% indoor and 8% outdoor when the same camera is used for training and testing. When the test camera is different, the improvement reaches 62%. Code available at: https://github.com/alexklwong/ calibrated-backprojection-network.

show abstract

An Adaptive Framework for Learning Unsupervised Depth Completion

Cited by 29 publications

References 32 publications

Learning Topology From Synthetic Data for Unsupervised Depth Completion

Learning Topology From Synthetic Data for Unsupervised Depth Completion

MetaComp: Learning to Adapt for Online Depth Completion

Unsupervised Depth Completion with Calibrated Backprojection Layers

Contact Info

Product

Resources

About