DepthFormer: Exploiting Long-range Correlation and Local Information for Accurate Monocular Depth Estimation

Li, Zhenyu; Chen, Zehui; Liu, Xianming; Jiang, Junjun

doi:10.1007/s11633-023-1458-0

Cited by 76 publications

(7 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors also integrate a revisited version of CutDepth data augmentation method [27] which is able to improve the training process on the NYU Depth v2 dataset without needing additional data. Li et al propose DepthFormer [6] and BinsFormer [28], where the first one is composed of a fully-transformer encoder and a convolutional decoder interleaved by an interaction module to enhance transformer encoded and CNN decoded features. Differently, in BinsFormer the idea of the authors is to use a multi-scale transformer decoder to generate adaptive bins and to recover spatial geometry information from the encoded features.…”

Section: B Vit-based Mde Methodsmentioning

confidence: 99%

SPEED: Separable Pyramidal Pooling EncodEr-Decoder for Real-Time Monocular Depth Estimation on Low-Resource Settings

et al. 2022

View full text Add to dashboard Cite

The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene understanding and visual odometry, which are key components in autonomous and robotic systems. Approaches based on the state of the art vision transformer architectures are extremely deep and complex not suitable for realtime inference operations on edge and autonomous systems equipped with low resources (i.e. robot indoor navigation and surveillance). This paper presents SPEED, a Separable Pyramidal pooling EncodEr-Decoder architecture designed to achieve real-time frequency performances on multiple hardware platforms. The proposed model is a fast-throughput deep architecture for MDE able to obtain depth estimations with high accuracy from low resolution images using minimum hardware resources (i.e. edge devices). Our encoderdecoder model exploits two depthwise separable pyramidal pooling layers, which allow to increase the inference frequency while reducing the overall computational complexity. The proposed method performs better than other fast-throughput architectures in terms of both accuracy and frame rates, achieving realtime performances over cloud CPU, TPU and the NVIDIA Jetson TX1 on two indoor benchmarks: the NYU Depth v2 and the DIML Kinect v2 datasets.INDEX TERMS Computer vision, monocular depth estimation, fast-throughput, edge devices

show abstract

Section: B Vit-based Mde Methodsmentioning

confidence: 99%

SPEED: Separable Pyramidal Pooling EncodEr-Decoder for Real-Time Monocular Depth Estimation on Low-Resource Settings

et al. 2022

View full text Add to dashboard Cite

show abstract

“…For the decoder, a local planar guidance (LPG) layer was proposed, which effectively establishes a direct and explicit relationship between the feature extracted from the encoder and the final output. In 2022, a paper by Zhenyu Li et al [ 12 ] sought to improve both the global and local features extracted from the encoder through the HAHI (hierarchical aggregation heterogeneous integration) module. The HAHI module consists of a self-attention module for the enhancement of features obtained from hierarchical layers of the Swin Transformer and a cross-attention module for affinity modeling of features obtained from two heterogeneous encoder branches.…”

Section: Related Workmentioning

confidence: 99%

A Novel Method for Monocular Depth Estimation Using an Hourglass Neck Module

Oh,

Lee

2024

Sensors

View full text Add to dashboard Cite

In this paper, we propose a novel method for monocular depth estimation using the hourglass neck module. The proposed method has the following originality. First, feature maps are extracted from Swin Transformer V2 using a masked image modeling (MIM) pretrained model. Since Swin Transformer V2 has a different patch size for each attention stage, it is easier to extract local and global features from images input by the vision transformer (ViT)-based encoder. Second, to maintain the polymorphism and local inductive bias of the feature map extracted from Swin Transformer V2, a feature map is input into the hourglass neck module. Third, deformable attention can be used at the waist of the hourglass neck module to reduce the computation cost and highlight the locality of the feature map. Finally, the feature map traverses the neck and proceeds through a decoder, comprised of a deconvolution layer and an upsampling layer, to generate a depth image. To evaluate the objective reliability of the proposed method in this paper, we used the NYU Depth V2 dataset to compare and evaluate the methods published in other papers. As a result of the experiment, the RMSE value of the novel method for monocular depth estimation using the hourglass neck module proposed in this paper was 0.274, which was lower than those published in other papers. The lower the RMSE value, the better the depth estimation method; therefore, its efficiency compared to other techniques has been proven.

show abstract

“…The authors in Ref. [19] use the encoder composed of the Transformer [20] branch and CNN branch to fully obtain long-distance correlation and local information, and multi-branch encoder leads to the complex model. The work in Ref.…”

Section: Related Wordmentioning

confidence: 99%

A monocular image depth estimation method based on weighted fusion and point‐wise convolution

Lei

Zhang

Sun

2023

IET Computer Vision

View full text Add to dashboard Cite

The existing monocular depth estimation methods based on deep learning have difficulty in estimating the depth near the edges of the objects in an image when the depth distance between these objects changes abruptly and decline in accuracy when an image has more noises. Furthermore, these methods consume more hardware resources because they have huge network parameters. To solve these problems, this paper proposes a depth estimation method based on weighted fusion and point‐wise convolution. The authors design a maximum‐average adaptive pooling weighted fusion module (MAWF) that fuses global features and local features and a continuous point‐wise convolution module for processing the fused features derived from the (MAWF) module. The two modules work closely together for three times to perform weighted fusion and point‐wise convolution of features of multi‐scale from the encoder output, which can better decode the depth information of a scene. Experimental results show that our method achieves state‐of‐the‐art performance on the KITTI dataset with δ1 up to 0.996 and the root mean square error metric down to 8% and has demonstrated the strong generalisation and robustness.

show abstract

DepthFormer: Exploiting Long-range Correlation and Local Information for Accurate Monocular Depth Estimation

Cited by 76 publications

References 51 publications

SPEED: Separable Pyramidal Pooling EncodEr-Decoder for Real-Time Monocular Depth Estimation on Low-Resource Settings

SPEED: Separable Pyramidal Pooling EncodEr-Decoder for Real-Time Monocular Depth Estimation on Low-Resource Settings

A Novel Method for Monocular Depth Estimation Using an Hourglass Neck Module

A monocular image depth estimation method based on weighted fusion and point‐wise convolution

Contact Info

Product

Resources

About