Dense Hybrid Recurrent Multi-view Stereo Net with Dynamic Consistency Checking

Detailed Facial Geometry Recovery from Multi-View Images by Learning an Implicit Function

“…Gu et al (Gu et al 2020) extends the prior volume-based MVS network where cost volume is built upon a feature pyramid encoding geometry and context at gradually finer scales. Yan et al (Yan et al 2020) proposes a dense hybrid recurrent multi-view stereo net with dynamic consistency checking for accurate dense point cloud reconstruction. Yi et al (Yi et al 2020) presents a pyramid multi-view stereo network with the self-adaptive view aggregation.…”

“…Surface-aware sampling. Comparing to CNN based regularization network in previous works (Yao et al 2018(Yao et al , 2019Gu et al 2020;Yi et al 2020;Yan et al 2020), the implicit function benefits from surface-aware sampling strategy, which uses more points close to the surface for training. This strategy has been used for human shape recovery in the 3D volumetric space (Saito et al 2019(Saito et al , 2020, while we extend it to sample in the UVD space with a Gaussian distribution.…”

“…where v is one of the n vertices in ground-truth mesh G, and u is one of the m vertices in predicted mesh M. We only measure the accuracy in the facial area, excluding eyes and inner mouth where the ground-truth geometry is hard to obtain. Considering the result meshes of generalized MVS methods (Yao et al 2018;Gu et al 2020;Yi et al 2020;Yan et al 2020) tend to extend out of the facial area, we mask out the outer mesh using the ground-truth mask of the frontal view. We run the experiments of comparison and ablation study on our testing sets, which contains 1341 tuples of data.…”

“…We compare our method with previous methods quantitatively and qualitatively. The baselines are MVS-Net (Yao et al 2018), CasMVSNet (Gu et al 2020), D2HC-RMVSNet (Yan et al 2020), and PVA-MVSNET (Yi et al 2020), which are state-of-the-art generalized MVS methods. We evaluate two models for these methods: 'ori' means the originally trained model provided by the authors, which were trained on DTU dataset (Aanaes et al 2016;Jensen et al 2014); 'ft' means the model fine-tuned using our training data generated from FaceScape dataset.…”

“…images. The key observation is that the previous state-ofthe-art learning-based methods (Yao et al 2018(Yao et al , 2019Chen et al 2019;Gu et al 2020;Yi et al 2020;Yan et al 2020) build and refine the cost volume in the camera frustum space, which is memory-consuming and limits the resolution. Though several works (Gu et al 2020;Riegler, Ulusoy, and Geiger 2017;Wang et al 2017) tried to improve the efficiency for volume-based neural network, they didn't utilize the facial prior and fail to recover the detailed geometry.…”