Unsupervised Depth Completion From Visual Inertial Odometry

Abstract

We describe a method to infer dense depth from camera motion and sparse depth as estimated using a visualinertial odometry system. Unlike other scenarios using point clouds from lidar or structured light sensors, we have few hundreds to few thousand points, insufficient to inform the topology of the scene. Our method first constructs a piecewise planar scaffolding of the scene, and then uses it to infer dense depth using the image along with the sparse points. We use a predictive cross-modal criterion, akin to "self-supervision," measuring photometric consistency across time, forward-backward pose consistency, and geometric compatibility with the sparse point cloud. We also launch the first visual-inertial + depth dataset, which we hope will foster additional exploration into combining the complementary strengths of visual and inertial sensors. To compare our method to prior work, we adopt the unsupervised KITTI depth completion benchmark, and show stateof-the-art performance on it.

Learning Topology From Synthetic Data for Unsupervised Depth Completion

“…Our goal is to recover a 3D scene from a real RGB image I t : Ω ⊂ R 2 → R 3 + and the associated set of sparse depth measurements z : Ω z ⊂ Ω → R + , without access to ground-truth depth annotations. We follow the unsupervised monocular training paradigm [21], [33] and assume there exists temporally adjacent frames, I τ for τ ∈ T . = {t − 1, t + 1} denoting the previous and the next time stamp relative to I t , available during training.…”

“…Unsupervised depth completion assumes additional (stereo, temporally consecutive frames) data available during training. Both stereo [26], [36] and monocular [21], [31], [32], [33] paradigms learn dense depth from an image and sparse depth measurements by minimizing the photometric error between the input image and its reconstruction from other views along with the difference between prediction and sparse depth input (sparse depth reconstruction). [21] used Perspective-n-Point [19] and RANSAC [9] to align consecutive frames and [34], [32] proposed an adaptive weighting framework.…”

“…To accomplish this, we leverage synthetic datasets with accurate dense depth to capture the patterns of complex geometry, present in everyday objects, that hand-crafted priors (e.g. nearest neighbor, piece-wise smoothness [33]) cannot.…”

“…The resulting points are subsampled using kmeans to produce the final 375 corners which corresponds to 0.49% of all the pixels. We train ScaffNet on SceneNet for indoor target dataset (VOID [33]). Despite SceneNet (indoor, household rooms) having different scene structures from VOID (indoor and outdoor, laboratories and gardens), ScaffNet trained on SceneNet generalizes to VOID.…”

“…The testing set contains 800 frames. We follow the evaluation protocol of [33] and cap the depths between 0.2 and 5 meters.…”