Attention meets Geometry: Geometry Guided Spatial-Temporal Attention for Consistent Self-Supervised Monocular Depth Estimation

Ruhkamp, Patrick; Gao, Daoyi; Chen, Hanzhi; Navab, Nassir; Busam, Benjamin

doi:10.1109/3dv53792.2021.00092

Cited by 24 publications

(5 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In 2021, Gao et al proposed a weakly supervised MDE method utilizing a commonly used U-net architecture [31]. Both encoder and decoder are constructed based on VGG blocks and channel self-attention modules to recalibrate the channel-wise information.…”

Section: Supervised Trainingmentioning

confidence: 99%

“…The low-resolution disparity is then processed by a multi-scale decoder to yield highresolution disparity. A temporally consistent depth (TC-Depth) prediction method was proposed by Ruhkamp et al [31], in which a spatial-temporal attention block is placed at the bottleneck to aggregate geometric and sequential consistencies. Unlike those of SE and CBAM, TC depth adopts physical distance between points of 3D space to measure the spatial attention map defined as follows:…”

Section: Self-supervised Trainingmentioning

confidence: 99%

See 1 more Smart Citation

Attention Mechanism Used in Monocular Depth Estimation: An Overview

Li,

Wei,

Fan

2023

Applied Sciences

View full text Add to dashboard Cite

Monocular depth estimation (MDE), as one of the fundamental tasks of computer vision, plays important roles in downstream applications such as virtual reality, 3D reconstruction, and robotic navigation. Convolutional neural networks (CNN)-based methods gained remarkable progress compared with traditional methods using visual cues. However, recent researches reveal that the performance of MDE using CNN could be degraded due to the local receptive field of CNN. To bridge the gap, various attention mechanisms were proposed to model the long-range dependency. Although reviews of MDE algorithms based on CNN were reported, a comprehensive outline of how attention boosts MDE performance is not explored yet. In this paper, we firstly categorize recent attention-related works into CNN-based, Transformer-based, and hybrid (CNN–Transformer-based) approaches in the light of how the attention mechanism impacts the extraction of global features. Secondly, we discuss the details and contributions of attention-based MDE methods published from 2020 to 2022. Then, we compare the performance of the typical attention-based methods. Finally, the challenges and trends of the attention mechanism used in MDE are discussed.

show abstract

Section: Supervised Trainingmentioning

confidence: 99%

Section: Self-supervised Trainingmentioning

confidence: 99%

Attention Mechanism Used in Monocular Depth Estimation: An Overview

Li,

Wei,

Fan

2023

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…In addition, several studies improve performance by adding additional semantic information or motion information learning networks [41]- [44]. In addition, multi-frame-based evaluation is performed to further utilize geometric information [2], [37], [42], [45], [46].…”

Section: B Self-supervised Monocular Depth Estimationmentioning

confidence: 99%

Self-Supervised Monocular Depth Estimation Using Hybrid Transformer Encoder

et al. 2022

View full text Add to dashboard Cite

Depth estimation using monocular camera sensors is an important technique in computer vision. Supervised monocular depth estimation requires a lot of data acquired from depth sensors. However, acquiring depth data is an expensive task. We sometimes cannot acquire data due to the limitations of the sensor. View synthesisbased depth estimation research is a self-supervised learning method that does not require depth data supervision. Previous studies mainly use CNN-based networks in encoders. CNN is suitable for extracting local features through convolution operation. Recent vision transformers are suitable for global feature extraction based on multi-self-attention modules. In this paper, we propose a hybrid network combining CNN and vision transformer network in self-supervised learning-based monocular depth estimation. We design an encoder-decoder structure that uses CNNs in the earlier stage of extracting local features and a vision transformer in the later stages of extracting global features. We evaluate the proposed network through various experiments based on KITTI and Cityscapes datasets. The results showed higher performance than previous studies and reduced parameters and computations. Codes and trained models are available at https://github.com/fogfog2/manydepthformer.

show abstract

“…3D geometric constraints [27,6,45] were proposed that penalize the euclidean distances between the reconstructed point clouds of two consecutive frames, after transforming one to the other. Similar geometric constraints on the depth maps were proposed [1,25,35] which minimize the inconsistency of the estimated disparity maps of two consecutive frames, after warping one onto the other. Our work lies in this category and differs in the fact that we propose complementary constraints on the pose, which can, in principle, be added and used together with the other temporal constraints.…”

Section: Self-supervised Temporal Consistencymentioning

confidence: 99%

“…Ruhkamp et al . [35] propose a strategy to detect and mask inconsistent regions such as occlusions, in neighboring depth maps via a cycle consistency of the reprojected RGB frames.…”

Section: Similar Constraints In Literaturementioning

confidence: 99%

Semantic and Optical Flow Guided Self-supervised Monocular Depth and Ego-Motion Estimation

Fang

Liu

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Self-supervised monocular depth estimation approaches suffer not only from scale ambiguity but also infer temporally inconsistent depth maps w.r.t. scale. While disambiguating scale during training is not possible without some kind of ground truth supervision, having scale consistent depth predictions would make it possible to calculate scale once during inference as a post-processing step and use it over-time. With this as a goal, a set of temporal consistency losses that minimize pose inconsistencies over time are introduced. Evaluations show that introducing these constraints not only reduces depth inconsistencies but also improves the baseline performance of depth and ego-motion prediction.

show abstract

Attention meets Geometry: Geometry Guided Spatial-Temporal Attention for Consistent Self-Supervised Monocular Depth Estimation

Cited by 24 publications

References 39 publications

Attention Mechanism Used in Monocular Depth Estimation: An Overview

Attention Mechanism Used in Monocular Depth Estimation: An Overview

Self-Supervised Monocular Depth Estimation Using Hybrid Transformer Encoder

Semantic and Optical Flow Guided Self-supervised Monocular Depth and Ego-Motion Estimation

Contact Info

Product

Resources

About