PA-MVSNet: Sparse-to-Dense Multi-View Stereo With Pyramid Attention

Zhang, Ke; Liu, Mengyu; Zhang, Jinlai; Dong, Zhenbiao

doi:10.1109/access.2021.3058522

Cited by 17 publications

(7 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, MVSTR [87], LANet [84] and TransMVSNet [14] unitize the attention mechanisms to extract dense features with global contexts. PA-MVSNet [82] and AACVP-MVSNet [78] introduce self-attention layers for hierarchical features extraction, which is able to capture multi-scale matching clues for the subsequent depth inference task. AttMVS [38] introduces an attentionenhanced matching confidence volume to improve the matching robustness.…”

Section: Related Workmentioning

confidence: 99%

RayMVSNet: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo

Shi²,

Wang

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Learning-based multi-view stereo (MVS) has by far centered around 3D convolution on cost volumes. Due to the high computation and memory consumption of 3D CNN, the resolution of output depth is often considerably limited. Different from most existing works dedicated to adaptive refinement of cost volumes, we opt to directly optimize the depth value along each camera ray, mimicking the range (depth) finding of a laser scanner. This reduces the MVS problem to ray-based depth optimization which is much more light-weight than full cost volume optimization. In particular, we propose RayMVSNet which learns sequential prediction of a 1D implicit field along each camera ray with the zero-crossing point indicating scene depth. This sequential modeling, conducted based on transformer features, essentially learns the epipolar line search in traditional multi-view stereo. We devise a multi-task learning for better optimization convergence and depth accuracy. We found the monotonicity property of the SDFs along each ray greatly benefits the depth estimation. Our method ranks top on both the DTU and the Tanks & Temples datasets over all previous learning-based methods, achieving an overall reconstruction score of 0.33mm on DTU and an F-score of 59.48% on Tanks & Temples. It is able to produce high-quality depth estimation and point cloud reconstruction in challenging scenarios such as objects/scenes with non-textured surface, severe occlusion, and highly varying depth range. Further, we propose RayMVSNet++ to enhance contextual feature aggregation for each ray through designing an attentional gating unit to select semantically relevant neighboring rays within the local frustum around that ray. This improves the performance on datasets with more challenging examples (e.g. low-quality images caused by poor lighting conditions or motion blur). RayMVSNet++ achieves state-of-the-art performance on the ScanNet dataset. In particular, it attains an AbsRel of 0.058m and produces accurate results on the two subsets of textureless regions and large depth variation.

show abstract

Section: Related Workmentioning

confidence: 99%

RayMVSNet: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo

Shi²,

Wang

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…Unlike the convolutional neural networks (CNN) [20,21] models' local image pattern, Transformer employs self-attention [22] on a tokenized image patch to model the context information of the image. In recent studies, pure Transformer architectures were introduced to many visual recognition and image classification tasks and achieved promising results.…”

Section: Transformermentioning

confidence: 99%

ARDformer: Agroforestry Road Detection for Autonomous Driving Using Hierarchical Transformer

Firkat

Zhang

et al. 2022

Sensors

Self Cite

View full text Add to dashboard Cite

Road detection is a crucial part of the autonomous driving system, and semantic segmentation is used as the default method for this kind of task. However, the descriptive categories of agroforestry are not directly definable and constrain the semantic segmentation-based method for road detection. This paper proposes a novel road detection approach to overcome the problem mentioned above. Specifically, a novel two-stage method for road detection in an agroforestry environment, namely ARDformer. First, a transformer-based hierarchical feature aggregation network is used for semantic segmentation. After the segmentation network generates the scene mask, the edge extraction algorithm extracts the trail’s edge. It then calculates the periphery of the trail to surround the area where the trail and grass are located. The proposed method is tested on the public agroforestry dataset, and experimental results show that the intersection over union is approximately 0.82, which significantly outperforms the baseline. Moreover, ARDformer is also effective in a real agroforestry environment.

show abstract

“…Another solution is the PAMVSNet (Zhang et al, 2021), which uses a pyramidal attention module, obtaining more in formation from the original image and generating a signifi cant improvement in the representation of features. This so lution has improved image quality and less noise, while it has an increase the execution time for the generation of multi pyramidal views.…”

Section: Related Workmentioning

confidence: 99%

Fast and Robust 3D Reconstruction Solution from Permissive Open-Source Code

Lyra¹,

Pinto²,

Lima³

et al. 2021

JIS

View full text Add to dashboard Cite

With the growth of access to faster computers and more powerful cameras, the 3D reconstruction of objects has become one of the public's main topics of research and demand. This task is vigorously applied in creating virtual environments, creating object models, and other activities. One of the techniques for obtaining 3D features is photogrammetry, mapping objects and scenarios using only images. However, this process is very costly and can be pretty time-consuming for large datasets. This paper proposes a robust, efficient reconstruction pipeline with a low runtime in batch processing and permissive code. It is even possible to commercialize it without the need to keep the code open. We mix an improved structure from motion algorithm and a recurrent multi-view stereo reconstruction. We also use the Point Cloud Library for normal estimation, surface reconstruction, and texture mapping. We compare our results with state-of-the-art techniques using benchmarks and our datasets. The results showed a decrease of 69.4% in the average execution time, with high quality but a greater need for more images to achieve complete reconstruction.

show abstract

PA-MVSNet: Sparse-to-Dense Multi-View Stereo With Pyramid Attention

Abstract: We thank all the participants of the study for their time and useful comments. We appreciate the support from our colleagues and classmates.

Cited by 17 publications

References 41 publications

RayMVSNet: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo

RayMVSNet: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo

ARDformer: Agroforestry Road Detection for Autonomous Driving Using Hierarchical Transformer

Fast and Robust 3D Reconstruction Solution from Permissive Open-Source Code

Contact Info

Product

Resources

About