Depth Prediction without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos

Abstract

Learning to predict scene depth from RGB inputs is a challenging task both for indoor and outdoor robot navigation. In this work we address unsupervised learning of scene depth and robot ego-motion where supervision is provided by monocular videos, as cameras are the cheapest, least restrictive and most ubiquitous sensor for robotics. Previous work in unsupervised image-to-depth learning has established strong baselines in the domain. We propose a novel approach which produces higher quality results, is able to model moving objects and is shown to transfer across data domains, e.g. from outdoors to indoor scenes. The main idea is to introduce geometric structure in the learning process, by modeling the scene and the individual objects; camera ego-motion and object motions are learned from monocular videos as input. Furthermore an online refinement method is introduced to adapt learning on the fly to unknown domains. The proposed approach outperforms all state-of-the-art approaches, including those that handle motion e.g. through learned flow. Our results are comparable in quality to the ones which used stereo as supervision and significantly improve depth prediction on scenes and datasets which contain a lot of object motion. The approach is of practical relevance, as it allows transfer across environments, by transferring models trained on data collected for robot navigation in urban scenes to indoor navigation settings. The code associated with this paper can be found at https://sites.google.com/ view/struct2depth.

DeepVIO: Self-supervised Deep Learning of Monocular Visual Inertial Odometry using 3D Geometric Constraints

“…The CNNs is trained by a photometric reconstruction loss, which was obtained by warping nearby views to the target using the computed depth and pose. In addition, [21] addresses unsupervised learning of scene depth, robot ego-motion and object motions where the supervision is provided by geometric structure of monocular videos as input. Furthermore, many recent efforts [11], [22] explore the geometric relationships between depth, camera motion, and flow for unsupervised learning of depth and flow estimation models.…”

Self-Supervised Learning With Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera

“…Vijayanarasimhan et al [41] additionally learn rigid motion parameters of multiple objects. Subsequent methods further proposed improvements based on various techniques, such as an ICP alignment loss [30], supervision from SfM algorithms [26], optical flow [48,52], edges [47], modeling multiple rigid motions informed by instance segmentation [3], motion segmentation [36], minimum projection loss [16], etc.…”

“…Frames that are similar to the test scenes are removed from the training set. We compare the performance of the proposed framework with the baseline, as well as recent stateof-the-art works in the same setting [3,30,42,47,48,51,52].…”

“…We notice that the online refinement is not used in other methods except [3]. Thus to facilitate the comparison, we also report the results of GLNet without online refinement, denoted as GLNet (-ref.).…”

“…We also conduct an ablation study over refinement strategies. We use the normal parameter finetuning without regularizer as baseline, as in [3]. We compare the baseline with the two proposed refinement strategies, i.e.…”