Monocular Depth Estimation Through Virtual-World Supervision and Real-World SfM Self-Supervision

Gurram, Akhil; Tuna, Ahmet Faruk; Shen, Fengyi; Urfalıoḡlu, Onay; López, Antonio

doi:10.1109/tits.2021.3117059

Cited by 29 publications

(4 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, we compared our method to others (see Table 5) and achieved a lower AbsRel on both KITTI and DDAD1 target domains [35,24] when using real source datatasets that were not specifically tailored to them, by only using a single depth-scaling factor. We also achieved competitive or better accuracy to other depth-scale-transfer methods that used specially tailored synthetic data for KITTI [26,24,35,41] and DDAD [24], and also showed that vKITTI2 can be successfully used as a source to other real datasets such as DDAD. Our accuracy is also competitive to weakly-supervised methods that used target GT velocity [21], GPS [7] or IMU [47] during training, and to online depth scaling methods [36,46] that require road visibility during inference.…”

Section: Transferring the Global Depth Scaling Factor From The Source...mentioning

confidence: 67%

“…Since real and synthetic data still differ in style, initial attempts focused on closing these gaps using style transfer from the target to the source domain [2,49,48]. Recent solutions trained on such synthetic and real datasets [26,24,35,41] used mixed-supervision, where the MDE was self-supervised using images from both domains and also fully-supervised using the source GT depth, thus tightly coupled between the depth ranking and scaling. In addition, some solutions used additional intermediate tasks such as semantic segmentation [24] or introduced an additional network to predict the depth-scale [41].…”

Section: Previous Workmentioning

confidence: 99%

See 1 more Smart Citation

One scalar is all you need -- absolute depth estimation using monocular self-supervision

Dana¹,

Carmel²,

Shomer³

et al. 2023

Preprint

View full text Add to dashboard Cite

Self-supervised monocular depth estimators can be trained or fine-tuned on new scenes using only images and no ground-truth depth data, achieving good accuracy. However, these estimators suffer from the inherent ambiguity of the depth scale, significantly limiting their applicability. In this work, we present a method for transferring the depth-scale from existing source datasets collected with ground-truth depths to depth estimators that are trained using self-supervision on a newly collected target dataset consisting of images only, solving a significant limiting factor.We show that self-supervision based on projective geometry results in predicted depths that are linearly correlated with their ground-truth depths. Moreover, the linearity of this relationship also holds when jointly training on images from two different (real or synthetic) source and target domains. We utilize this observed property and model the relationship between the ground-truth and the predicted up-to-scale depths of images from the source domain using a single global scalar. Then, we scale the predicted up-to-scale depths of images from the target domain using the estimated global scaling factor, performing depth-scale transfer between the two domains. This suggested method was evaluated on the target KITTI and DDAD datasets, while using other real or synthetic source datasets, that have a larger field-of-view, other image style or structural content. Our approach achieves competitive accuracy on KITTI, even without using the specially tailored vKITTI or vKITTI2 datasets, and higher accuracy on DDAD, when using both real or synthetic source datasets.

show abstract

Section: Transferring the Global Depth Scaling Factor From The Source...mentioning

confidence: 67%

Section: Previous Workmentioning

confidence: 99%

One scalar is all you need -- absolute depth estimation using monocular self-supervision

Dana¹,

Carmel²,

Shomer³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Yu 9 harryjun@ustc.edu.cn M. Jing 9 jing mohan@mail.ustc.edu.cn X. Qi 9 xiaohua000109@163.com Network. Predictions were obtained as a mixture of multiple networks: DiffNet [107], FeatDepth [74] and MonoDEVS-Net [33]. DiffNet and FeatDepth used a ResNet backbone, while MonoDEVSNet used DenseNet [38].…”

Section: Challenge Submissionsmentioning

confidence: 99%

2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)

2023

View full text Add to dashboard Cite

This paper discusses the results for the second edition of the Monocular Depth Estimation Challenge (MDEC). This edition was open to methods using any form of supervision, including fully-supervised, self-supervised, multi-task or proxy depth. The challenge was based around the SYNS-Patches dataset, which features a wide diversity of environments with high-quality dense ground-truth. This includes complex natural environments, e.g. forests or fields, which are greatly underrepresented in current benchmarks.The challenge received eight unique submissions that outperformed the provided SotA baseline on any of the pointcloud-or image-based metrics. The top supervised submission improved relative F-Score by 27.62%, while the top self-supervised improved it by 16.61%. Supervised submissions generally leveraged large collections of datasets to improve data diversity. Self-supervised submissions instead updated the network architecture and pretrained backbones. These results represent a significant progress in the field, while highlighting avenues for future research, such as reducing interpolation artifacts at depth boundaries, improving self-supervised indoor performance and overall natural image accuracy.

show abstract

“…Therefore, it can operate on depth maps Fig. 1: FusionLoc architecture diagram generated using stereo depth estimation techniques [12], [13], monocular depth estimation [14], [15], and infrared cameras such as the ones found in Kinect sensors.…”

Section: Introductionmentioning

confidence: 99%

Deep Camera Pose Regression Using Pseudo-LiDAR

Raza¹,

Lolic²,

Akhter³

et al. 2022

Preprint

View full text Add to dashboard Cite

An accurate and robust large-scale localization system is an integral component for active areas of research such as autonomous vehicles and augmented reality. To this end, many learning algorithms have been proposed that predict 6DOF camera pose from RGB or RGB-D images. However, previous methods that incorporate depth typically treat the data the same way as RGB images, often adding depth maps as additional channels to RGB images and passing them through convolutional neural networks (CNNs). In this paper, we show that converting depth maps into pseudo-LiDAR signals, previously shown to be useful for 3D object detection, is a better representation for camera localization tasks by projecting point clouds that can accurately determine 6DOF camera pose. This is demonstrated by first comparing localization accuracies of a network operating exclusively on pseudo-LiDAR representations, with networks operating exclusively on depth maps. We then propose FusionLoc, a novel architecture that uses pseudo-LiDAR to regress a 6DOF camera pose. FusionLoc is a dual stream neural network, which aims to remedy common issues with typical 2D CNNs operating on RGB-D images. The results from this architecture are compared against various other state-of-the-art deep pose regression implementations using the 7 Scenes dataset. The findings are that FusionLoc performs better than a number of other camera localization methods, with a notable improvement being, on average, 0.33m and 4.35 • more accurate than RGB-D PoseNet. By proving the validity of using pseudo-LiDAR signals over depth maps for localization, there are new considerations when implementing large-scale localization systems.

show abstract

Monocular Depth Estimation Through Virtual-World Supervision and Real-World SfM Self-Supervision

Cited by 29 publications

References 50 publications

One scalar is all you need -- absolute depth estimation using monocular self-supervision

One scalar is all you need -- absolute depth estimation using monocular self-supervision

2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)

Deep Camera Pose Regression Using Pseudo-LiDAR

Contact Info

Product

Resources

About