Real-Time Monocular Depth Estimation Using Synthetic Data with Domain Adaptation via Image Style Transfer

Abstract

Monocular depth estimation using learning-based approaches has become promising in recent years. However, most monocular depth estimators either need to rely on large quantities of ground truth depth data, which is extremely expensive and difficult to obtain, or predict disparity as an intermediary step using a secondary supervisory signal leading to blurring and other artefacts. Training a depth estimation model using pixel-perfect synthetic data can resolve most of these issues but introduces the problem of domain bias. This is the inability to apply a model trained on synthetic data to real-world scenarios. With advances in image style transfer and its connections with domain adaptation (Maximum Mean Discrepancy), we take advantage of style transfer and adversarial training to predict pixel perfect depth from a single real-world color image based on training over a large corpus of synthetic environment data. Experimental results indicate the efficacy of our approach compared to contemporary state-of-the-art techniques.

Progressive Fusion for Unsupervised Binocular Depth Estimation Using Cycled Networks

“…However, supervised learning-based approaches rely on expensive ground-truth depth data for training and are not flexible to be deployed in novel environments. Even if synthetic data generation has been proposed to partially tackle this issue [26], the cost of synthesizing realistic data remains high.…”

To Complete or to Estimate, That is the Question: A Multi-Task Approach to Depth Completion and Monocular Depth Estimation

“…The primary reason for using synthetic images [17] during training is that despite the increased depth density of the real-world imagery [54], depth information for the majority of the scene is still missing, leading to undesirable artefacts in regions where depth values are not available. A naïve solution would be to only use synthetic data to resolve the issue, but due to differences in the data domains, a model only trained on synthetic data cannot be expected to perform well on real-world images without domain adaptation [5,63]. Consequently, we opt for randomly sampling training images from both datasets to force the overall model to capture the underlying distribution of both data domains, and therefore, learn the full dense structure of a synthetic scene while simultaneously modelling the contextual complexity of the naturally-sensed real-world images.…”

“…• A joint multi-task framework for depth prediction encouraging improved geometric and contextual learning to boost performance. • Monocular depth estimation via adversarial training, a deep architecture with skip connections and a robust compound objective function directly supervised using this framework to outperform prior contemporary work [5,7,14,20,31,36,62,66]. • Sparse to dense depth completion via the same multitask model, capable of generating a dense depth output given a sparse depth input captured via a LiDAR sensor with results superior to prior contemporary work [10,16,40,50,54].…”

“…In recent years, monocular depth estimation (i.e. estimating scene depth from a single RGB image) has received widespread attention within both academia and industry as a more effective, economical and innovative alternative to more conventional depth estimation strategies [5,6,14,18,20,31,66].…”

Counterfactual Depth from a Single RGB Image

“…Markov random field (MRF) [38] and Conditional random field (CRF) [31] can be applied to regress image depth against monocular images. More recent approaches use deep neural networks with multi-scale predictions [11,12], large-scale datasets [26,2] and user interactions [37]. Stereo provides strong cues for unsupervised learning [14,46] or semi-supervised learning with LiDAR [24].…”