Recurrent MVSNet for High-Resolution Multi-View Stereo Depth Inference

Abstract

Deep learning has recently demonstrated its excellent performance for multi-view stereo (MVS). However, one major limitation of current learned MVS approaches is the scalability: the memory-consuming cost volume regularization makes the learned MVS hard to be applied to highresolution scenes. In this paper, we introduce a scalable multi-view stereo framework based on the recurrent neural network. Instead of regularizing the entire 3D cost volume in one go, the proposed Recurrent Multi-view Stereo Network (R-MVSNet) sequentially regularizes the 2D cost maps along the depth direction via the gated recurrent unit (GRU). This reduces dramatically the memory consumption and makes high-resolution reconstruction feasible. We first show the state-of-the-art performance achieved by the proposed R-MVSNet on the recent MVS benchmarks. Then, we further demonstrate the scalability of the proposed method on several large-scale scenarios, where previous learned approaches often fail due to the memory constraint. Code is available at https://github.com/ YoYo000/MVSNet.

MVS2: Deep Unsupervised Multi-View Stereo with Multi-View Symmetry

“…Im et al [24] applied a plane sweeping approach to build a cost volume from deep features, then regularized the cost volume via a context-aware aggregation to improve depth regression. Very recently, Yao et al [31] introduced a scalable MVS framework based on the recurrent neural network to reduce the memory-consuming. Unsupervised Geometric Learning: Unsupervised learning has been developed in monocular depth estimation and binocular stereo matching by exploiting the photometric consistency and regularization.…”

“…Recently, the success of deep convolutional neural networks (CNNs) in monocular depth estimation [18,9,19] and binocular depth estimation [33,34] has been extended to MVS. Existing deep CNNs based MVS approaches [30,31,11,24] tend to represent MVS as an end-to-end regression problem. By exploiting large-scale ground truth 3D training data, these methods outperform traditional geometrybased approaches and dominate the leading boards on different benchmarking datasets [30,31].…”

“…Existing deep CNNs based MVS approaches [30,31,11,24] tend to represent MVS as an end-to-end regression problem. By exploiting large-scale ground truth 3D training data, these methods outperform traditional geometrybased approaches and dominate the leading boards on different benchmarking datasets [30,31]. However, the success of these supervised MVS approaches strongly depends on the availability of large-scale ground-truth 3D training data, which not only not always available but also may further hinder their generalization ability in never-seen-before open-world scenarios [34].…”

“…Our network structure differs from existing MVS and simple extension of unsupervised binocular stereo matching in the following aspects: a) Our network is symmetric to all the views, i.e., it treats each view equivalently and predicts the depth map for each view simultaneously. Existing supervised learning based MVS methods [30,31,11,27] apply an "asymmetric" design and infer depth map for the reference image only. Thus, multiple depth maps estimated from different viewpoints do not comply with the same 3D geometry and 3D point clouds processing is required to derive a consistent 3D geometry.…”

Flow-Motion and Depth Network for Monocular Stereo and Beyond

“…Given multiple images with known camera poses and intrinsic calibration, DeepMVS [10] generates cost volumes using learned feature maps and then estimates the disparity map by fusing multiple cost volumes. MVDepthNet [11], DPSNet [12] and MVSNet [13], [14] solve the same reconstruction problem but differ in the calculation of cost volumes and the structure of networks. On the other hand, given an RGB-D keyframe, DeepTAM [15] incrementally tracks the pose of a camera using synthetic viewpoints and can further estimate the depth map of the tracked frame.…”