Attention-based Multi-View Stereo Network

Chen, Liufeng; Zhai, Ruifang; Yang, Wanneng

doi:10.1145/3503961.3503978

Cited by 5 publications

(8 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MVSNet [42] applies a 3D CNN on a plane-swept cost volume at the reference view for depth estimation, achieving high-quality 3D reconstruction that outperforms classical traditional methods [13,32]. Following works have extended this technique with recurrent plane sweeping [43], point-based densification [7], confidence-based aggregation [26], and multiple cost volumes [8,15], improving the reconstruction quality. We propose to combine the cost-volume based deep MVS technique with differentiable volume rendering, enabling efficient reconstruction of radiance fields for neural rendering.…”

Section: Related Workmentioning

confidence: 99%

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

Chen¹,

Xu²,

Zhao³

et al. 2021

Preprint

View full text Add to dashboard Cite

b) MVS-NeRF no fine-tuning c) MVS-NeRF 6 min fine-tuning d) NeRF 9.5h optimization a) Source views PSNR: 16.63 PSNR: 25.96 PSNR: 23.36 * Equal contribution Research done when Anpei Chen was in a remote internship with UCSD.dense images are captured, our estimated radiance field representation can be easily fine-tuned; this leads to fast per-scene reconstruction with higher rendering quality and substantially less optimization time than NeRF.

show abstract

Section: Related Workmentioning

confidence: 99%

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

Chen¹,

Xu²,

Zhao³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Earlier work in this area uses CNN's for two-view [121] and multi-view stereo [33]. Lately, the learning-based MVS rely on the construction of 3D cost volume and use the deep neural networks for regularization and depth regression [18,38,113,46,71,114,111]. As most of these approaches utilize 3D CNN for cost volume regularization -which in general is computationally expensive, the majority of the recent work is motivated to meet the computational requirement with it.…”

Section: Related Workmentioning

confidence: 99%

“…Few methods attempt to address it by down-sampling the input [114,111]. Other attempts to improve the computational requirements uses sequential processing of cost volume [115], cascade of 3D cost volumes [19,32,112,105], small cost volume with point-based refinement [18], sparse cost volume with RGB and 2D CNN to densify the result [120], learning-based patch-wise matching [71,105] with RGB guided depth map super-resolution [105].…”

Section: Related Workmentioning

confidence: 99%

Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo

Kaya¹,

Kumar²,

Sarno³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present a modern solution to the multi-view photometric stereo problem (MVPS). Our work suitably exploits the image formation model in a MVPS experimental setup to recover the dense 3D reconstruction of an object from images. We procure the surface orientation using a photometric stereo (PS) image formation model and blend it with a multi-view neural radiance field representation to recover the object's surface geometry. Contrary to the previous multi-staged framework to MVPS, where the position, isodepth contours, or orientation measurements are estimated independently and then fused later, our method is simple to implement and realize. Our method performs neural rendering of multi-view images while utilizing surface normals estimated by a deep photometric stereo network. We render the MVPS images by considering the object's surface normals for each 3D sample point along the viewing direction rather than explicitly using the density gradient in the volume space via 3D occupancy information. We optimize the proposed neural radiance field representation for the MVPS setup efficiently using a fully connected deep network to recover the 3D geometry of an object. Extensive evaluation on the DiLiGenT-MV benchmark dataset shows that our method performs better than the approaches that perform only PS or only multi-view stereo (MVS) and provides comparable results against the state-of-the-art multistage fusion methods.

show abstract

“…The learned representation shows more robustness to low-texture regions and various lightings [22,45,47,7,32].…”

Section: Classical Stereo Matchingmentioning

confidence: 99%

“…Multi-view stereo (MVS) is one of the most fundamental problems in computer vision and has been studied over decades. Recently, learning-based MVS methods have witnessed significant improvement against their traditional counterparts [45,23,47,7]. In general, these methods formulate the task as an optimization problem, where the target is to minimize the overall summation of pixelwise depth discrepancy.…”

Section: Introductionmentioning

confidence: 99%

Normal Assisted Stereo Depth Estimation

Kusupati

Chen

2019

Preprint

View full text Add to dashboard Cite

Accurate stereo depth estimation plays a critical role in various 3D tasks in both indoor and outdoor environments. Recently, learning-based multi-view stereo methods have demonstrated competitive performance with limited number of views. However, in challenging scenarios, especially when building cross-view correspondences is hard, these methods still cannot produce satisfying results. In this paper, we study how to enforce the consistency between surface normal and depth at training time to improve the performance. We couple the learning of a multiview normal estimation module and a multi-view depth estimation module. In addition, we propose a novel consistency loss to train an independent consistency module that refines the depths from depth/normal pairs. We find that the joint learning can improve both the prediction of normal and depth, and the accuracy & smoothness can be further improved by enforcing the consistency. Experiments on MVS, SUN3D, RGBD and Scenes11 demonstrate the effectiveness of our method and state-ofthe-art performance.

show abstract

Attention-based Multi-View Stereo Network

Cited by 5 publications

References 13 publications

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo

Normal Assisted Stereo Depth Estimation

Contact Info

Product

Resources

About