Deep Laparoscopic Stereo Matching with Transformers

Cheng, Xuelian; Zhong, Yiran; Harandi, Mehrtash; Drummond, Tom; Wang, Zhiyong; Ge, Zongyuan

doi:10.1007/978-3-031-16449-1_44

Cited by 22 publications

(31 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DL-based algorithms have attracted great attention due to their superior performances in benchmark testing (Geiger et al, 2012). Depending on the task of learning, DL stereo methods can be further categorised into learning-based cost metrics (Žbontar & LeCun, 2015(Žbontar & LeCun, , 2016 and E2E learning (Chang & Chen, 2018;Cheng et al, 2020;Kendall et al, 2017;Xu & Zhang, 2020;. Learning-based cost metric was first introduced by Žbontar and LeCun (2015) to learn similarities from image patches.…”

Section: Rel Ated Work S Stereo Dim Algorithmsmentioning

confidence: 99%

“…The underlying concept is that these neural networks can directly capture more global features, hence, they may better perform (Chang & Chen, 2018;Cheng et al, 2020;Kendall et al, 2017). In addition to these intensively studied methods, there are a few methods that perform context learning for part of the traditional pipeline but do not fully fall into either of these DL categories, for example, SGM-Net (Seki & Pollefeys, 2017) learns the per-pixel smoothness penalty and GA-Net learns networks to guide the cost-aggregation process.…”

Section: Rel Ated Work S Stereo Dim Algorithmsmentioning

confidence: 99%

“…They have become a popular line of stereo DIM algorithms because they can directly predict highly accurate disparity maps through learning from geometry and context (e.g., cues such as shading, illumination, objects, etc.) rather than low-level features (Chang & Chen, 2018;Cheng et al, 2020;Gu et al, 2020;Kendall et al, 2017;Xu & Zhang, 2020;. Examples of this type of work are Geometry and Context Network (GCNet), Pyramid Stereo Matching Network (PSMNet), and LEAStereo, which due to their high accessibility and performances (i.e., winning the best rank in the KITTI 2012 and 2015 leaderboards (Chang & Chen, 2018;Cheng et al, 2020;Kendall et al, 2017)), have become popular in the field.…”

Section: Introductionmentioning

confidence: 99%

“…The one algorithm in this category is MC-CNN, which applies a Siamese network to learn the similarities (Žbontar & LeCun, 2015). Despite the very many E2E methods in the CV community (Laga, 2019), we choose three state-of-the-art (SOTA) methods that are frequently used by the community, and well-performed in the leaderboard (Chang & Chen, 2018;Cheng et al, 2020;Kendall et al, 2017), and have well-organised codes available. To achieve a comprehensive evaluation and analysis, we use nine satellite datasets from different locations, and each dataset contains ~100-500 stereopairs with their respective ground-truth lidar data.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A comparative study on deep‐learning methods for dense image matching of multi‐angle and multi‐date remote sensing stereo‐images

Albanwan

Qin

2022

The Photogrammetric Record

View full text Add to dashboard Cite

Deep-learning (DL) stereomatching methods gained great attention in remote sensing satellite datasets. However, most of these existing studies conclude assessments based only on a few/single stereo-images lacking a systematic evaluation on how robust DL methods are on satellite stereo-images with varying radiometric and geometric configurations. This paper provides an evaluation of four DL stereomatching methods through hundreds of multi-date multi-site satellite stereopairs with varying geometric configurations, against the traditional well-practiced Census-semi-global matching (SGM), to comprehensively understand their accuracy, robustness, generalisation capabilities, and their practical potential. The DL methods include a learning-based cost metric through convolutional neural networks (MC-CNN) followed by SGM, and three end-to-end (E2E) learning models using Geometry and Context Network (GCNet), Pyramid Stereo Matching Network (PSMNet), and LEAStereo. Our experiments show that E2E algorithms can achieve upper limits of geometric accuracies, while may not generalise well for unseen data.The learning-based cost metric and Census-SGM are rather robust and can consistently achieve acceptable results.

show abstract

Section: Rel Ated Work S Stereo Dim Algorithmsmentioning

confidence: 99%

Section: Rel Ated Work S Stereo Dim Algorithmsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A comparative study on deep‐learning methods for dense image matching of multi‐angle and multi‐date remote sensing stereo‐images

Albanwan

Qin

2022

The Photogrammetric Record

View full text Add to dashboard Cite

show abstract

“…It indicates that simultaneously considering Query-Key-Value is beneficial for discriminating feature importance more comprehensively and appropriately. Sharing the fixation ratio can trigger common feature representations like the weight sharing technique [57,85,100,12], and is also proven to be useful in improving the Top-1 accuracy in both Query-Key and Query-Key-Value cooperation cases. Moreover, it saves ∼0.1M parameters owing to the shared weights in learning the fixation ratio.…”

Section: Ablation Study and Model Analysismentioning

confidence: 99%

Linear Video Transformer with Feature Fixation

Lu¹,

Liu²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Vision Transformers have achieved impressive performance in video classification, while suffering from the quadratic complexity caused by the Softmax attention mechanism. Some studies alleviate the computational costs by reducing the number of tokens in attention calculation, but the complexity is still quadratic. Another promising way is to replace Softmax attention with linear attention, which owns linear complexity but presents a clear performance drop. We find that such a drop in linear attention results from the lack of attention concentration on critical features. Therefore, we propose a feature fixation module to reweight feature importance of the query and key before computing linear attention. Specifically, we regard the query, key, and value as various latent representations of the input token, and learn the feature fixation ratio by aggregating Query-Key-Value information. This is beneficial for measuring the feature importance comprehensively. Furthermore, we enhance the feature fixation by neighborhood association, which leverages additional guidance from spatial and temporal neighbouring tokens. The proposed method significantly improves the linear attention baseline and achieves state-of-the-art performance among linear video Transformers on three popular video classification benchmarks. With fewer parameters and higher efficiency, our performance is even comparable to some Softmax-based quadratic Transformers.

show abstract

Self-adapting Confidence Estimation for Stereo

Poggi

Aleotti

Tosi

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Project page: https://fedstereo.github.io/ ~80 FPS 13.86% D1-all 9.16% D1-all 8.36% D1-all ~10 FPS ~80 FPS (a) (b) (c) (d) Figure 1. Federated adaptation in challenging environments. When facing a domain very different from those observed during training -e.g., nighttime images (a) -stereo models [55] suffer drops in accuracy (b). By enabling online adaptation [41] (c) the network can improve its predictions, at the expense of decimating the framerate. In our federated framework, the model can demand the adaptation process to the cloud, to enjoy its benefits while maintaining the original processing speed (d).

show abstract

Deep Laparoscopic Stereo Matching with Transformers

Cited by 22 publications

References 26 publications

A comparative study on deep‐learning methods for dense image matching of multi‐angle and multi‐date remote sensing stereo‐images

A comparative study on deep‐learning methods for dense image matching of multi‐angle and multi‐date remote sensing stereo‐images

Linear Video Transformer with Feature Fixation

Self-adapting Confidence Estimation for Stereo

Contact Info

Product

Resources

About