Distilled Semantics for Comprehensive Scene Understanding from Videos

Tosi, Fabio; Aleotti, Filippo; Ramirez, Pierluigi Zama; Poggi, Matteo; Salti, Samuele; Stefano, Luigi Di; Mattoccia, Stefano

doi:10.1109/cvpr42600.2020.00471

Cited by 75 publications

(40 citation statements)

References 69 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On both monocular depth estimation and monocular visual odometry by the unsupervised learning of depth and pose, our method outperforms recent state-of-the-art methods. Besides, our method does not need auxiliary tasks, such as optical flow estimation [9]- [12], [26], [39], semantic segmentation [39] or dynamic mask estimation [11], [26], normal estimation [8]. This paper realizes end-to-end iterative view synthesis and pose refinement to jointly optimize the pose and depth estimation networks, allowing the overall parameters to selflearn according to the optimization objective.…”

Section: B Evaluation Resultsmentioning

confidence: 99%

3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video

Wang¹,

Jiquan²,

Zhao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Depth and ego-motion estimations are essential for the localization and navigation of autonomous robots and autonomous driving. Recent studies make it possible to learn the per-pixel depth and ego-motion from the unlabeled monocular video. A novel unsupervised training framework is proposed with 3D hierarchical refinement and augmentation using explicit 3D geometry. In this framework, the depth and pose estimations are hierarchically and mutually coupled to refine the estimated pose layer by layer. The intermediate view image is proposed and synthesized by warping the pixels in an image with the estimated depth and coarse pose. Then, the residual pose transformation can be estimated from the new view image and the image of the adjacent frame to refine the coarse pose. The iterative refinement is implemented in a differentiable manner in this paper, making the whole framework optimized uniformly. Meanwhile, a new image augmentation method is proposed for the pose estimation by synthesizing a new view image, which creatively augments the pose in 3D space but gets a new augmented 2D image. The experiments on KITTI demonstrate that our depth estimation achieves state-of-the-art performance and even surpasses recent approaches that utilize other auxiliary tasks. Our visual odometry outperforms all recent unsupervised monocular learning-based methods and achieves competitive performance to the geometrybased method, ORB-SLAM2 with back-end optimization.

show abstract

Section: B Evaluation Resultsmentioning

confidence: 99%

3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video

Wang¹,

Jiquan²,

Zhao³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, deep learningbased methods have dominated this field because of their powerful feature representation capacity. There are roughly two types of deep learning-based MDE, supervised methods [10], [11], [12], [13] and self-supervised (unsupervised) methods [14], [15], [16], [17], [18], [18], [19], [20], [21], [22], [23], [24], [25].…”

Section: Related Workmentioning

confidence: 99%

THE Benchmark: Transferable Representation Learning for Monocular Height Estimation

Xiong¹,

Huang²,

Hu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Generating 3D city models rapidly is crucial for many applications. Monocular height estimation is one of the most efficient and timely ways to obtain large-scale geometric information. However, existing works focus primarily on training and testing models using unbiased datasets, which don't align well with real-world applications. Therefore, we propose a new benchmark dataset to study the transferability of height estimation models in a cross-dataset setting. To this end, we first design and construct a large-scale benchmark dataset for cross-dataset transfer learning on the height estimation task. This benchmark dataset includes a newly proposed large-scale synthetic dataset, a newly collected real-world dataset, and four existing datasets from different cities. Next, two new experimental protocols, zero-shot and few-shot cross-dataset transfer, are designed. For few-shot cross-dataset transfer, we enhance the window-based Transformer with the proposed scale-deformable convolution module to handle the severe scale-variation problem. To improve the generalizability of deep models in the zero-shot cross-dataset setting, a max-normalization-based Transformer network is designed to decouple the relative height map from the absolute heights. Experimental results have demonstrated the effectiveness of the proposed methods in both the traditional and cross-dataset transfer settings. The datasets and codes are publicly available at https://thebenchmarkh.github.io/.

show abstract

“…As camera parameters are learnt in a fully self-supervised manner, even image sequences from wild can be used, where the network generalizes better. Later, other works, applicable to image sequences from wild, such as in [8] and [51] were proposed which followed an approach similar to [22] in camera parameters estimation. Our approach is similar to these, but does not need object motion mask as input [22,51] or online refinement [8], to output accurate depth maps, while assuming a pinhole camera model with minimum or no distortion.…”

Section: Related Workmentioning

confidence: 99%

“…This eliminates the plausible usage of potential data from the wild for training. Previous works [8,22,51] have addressed such issues in the past, however, are not on par with the other approaches which use ground-truth calibrated data. In our work, we primarily address this issue by removing the necessity of pre-calibrated data and also focus on refining monocular depth estimation accuracy.…”

Section: Introductionmentioning

confidence: 99%

CamLessMonoDepth: Monocular Depth Estimation with Unknown Camera Parameters

Chanduri¹,

Suri²,

Vozniak³

et al. 2021

Preprint

View full text Add to dashboard Cite

Perceiving 3D information is of paramount importance in many applications of computer vision. Recent advances in monocular depth estimation have shown that gaining such knowledge from a single camera input is possible by training deep neural networks to predict inverse depth and pose, without the necessity of ground truth data. The majority of such approaches, however, require camera parameters to be fed explicitly during training. As a result, image sequences from wild cannot be used during training. While there exist methods which also predict camera intrinsics, their performance is not on par with novel methods taking camera parameters as input. In this work, we propose a method for implicit estimation of pinhole camera intrinsics along with depth and pose, by learning from monocular image sequences alone. In addition, by utilizing efficient subpixel convolutions, we show that high fidelity depth estimates can be obtained. We also embed pixel-wise uncertainty estimation into the framework, to emphasize the possible applicability of this work in practical domain. Finally, we demonstrate the possibility of accurate prediction of depth information without prior knowledge of camera intrinsics, while outperforming the existing state-of-the-art approaches on KITTI benchmark.

show abstract

Distilled Semantics for Comprehensive Scene Understanding from Videos

Cited by 75 publications

References 69 publications

3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video

3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video

THE Benchmark: Transferable Representation Learning for Monocular Height Estimation

CamLessMonoDepth: Monocular Depth Estimation with Unknown Camera Parameters

Contact Info

Product

Resources

About