SMUDLP: Self-Teaching Multi-Frame Unsupervised Endoscopic Depth Estimation with Learnable Patchmatch

Abstract

Unsupervised monocular trained depth estimation models make use of adjacent frames as a supervisory signal during the training phase. However, temporally correlated frames are also available at inference time for many clinical applications, e.g., surgical navigation. The vast majority of monocular systems do not exploit this valuable signal that could be deployed to enhance the depth estimates. Those that do, achieve only limited gains due to the unique challenges in endoscopic scenes, such as low and homogeneous textures and inter-frame brightness fluctuations. In this work, we present SMUDLP, a novel and unsupervised paradigm for multi-frame monocular endoscopic depth estimation. The SMUDLP integrates a learnable patchmatch module to adaptively increase the discriminative ability in low-texture and homogeneous-texture regions, and enforces cross-teaching and self-teaching consistencies to provide efficacious regularizations towards brightness fluctuations. Our detailed experiments on both SCARED and Hamlyn datasets indicate that the SMUDLP exceeds state-of-the-art competitors by a large margin, including those that use single or multiple frames at inference time. The source code and trained models will be publicly available upon the acceptance.

Crafting Monocular Cues and Velocity Guidance for Self-Supervised Multi-Frame Depth Learning

“…The self-supervised method (Watson et al 2021) removes the requirement for human annotations and depth supervision, and it proposes a teacher-student training architecture to encourage the network to ignore unreliable regions in MVS cost volume. (Feng et al 2022) further improves the depth accuracy in regions of dynamic objects by disentangling object motions, and (Shao et al 2022) utilizes Deformable Convolution Networks (DCNs (Dai et al 2017)) to enhance the depth estimates in low-texture and homogeneous-texture regions. However, these methods enforce consistency between MVS depth and monocular depth, which underuses the geometric reasoning of MVS.…”

“…However, MVS is still challenged by unsatisfactory reconstructions in real-world scenes with non-Lambertian surfaces, textureless areas, and moving objects (Knapitsch et al 2017;Schöps et al 2017). To tackle these problems, teacherstudent training architectures (Watson et al 2021;Feng et al 2022;Shao et al 2022) are proposed to enforce consistency between monocular depth and MVS depth. However, the consistency pushes MVS depth to mimic monocular depth, which underuses the multi-view geometry, thus the performance of these methods is limited.…”