Detailed Facial Geometry Recovery from Multi-View Images by Learning an Implicit Function

Abstract

Recovering detailed facial geometry from a set of calibrated multi-view images is valuable for its wide range of applications. Traditional multi-view stereo (MVS) methods adopt an optimization-based scheme to regularize the matching cost. Recently, learning-based methods integrate all these into an end-to-end neural network and show superiority of efficiency. In this paper, we propose a novel architecture to recover extremely detailed 3D faces within dozens of seconds. Unlike previous learning-based methods that regularize the cost volume via 3D CNN, we propose to learn an implicit function for regressing the matching cost. By fitting a 3D morphable model from multi-view images, the features of multiple images are extracted and aggregated in the mesh-attached UV space, which makes the implicit function more effective in recovering detailed facial shape. Our method outperforms SOTA learning-based MVS in accuracy by a large margin on the FaceScape dataset. The code and data are released in https://github.com/zhuhao-nju/mvfr.

RAFaRe: Learning Robust and Accurate Non-parametric 3D Face Reconstruction from Pseudo 2D&3D Pairs

“…Datasets play a crucial role in the task of learning-based SVFR because their quality decides the performance of neural networks, while current public datasets cannot meet the demands. High-quality scans or multi-view reconstruction systems provide detailed and accurate shapes but lack generalization due to the complexity of the capture system (Xiao et al 2022). On the other hand, in-the-wild datasets (Zhu et al 2016) with optimization-based 3DMM fitting satisfy generality but introduce misalignment between 3D models and 2D images.…”

SimpModeling: Sketching Implicit Field to Guide Mesh Modeling for 3D Animalmorphic Head Design

“…Creating visually plausible 3D avatars is a long-standing computer graphics and vision problem. Compared with 3D face reconstruction methods, which take multi-view [26], [27] or monocular video [28], [29] as input, single image reconstruction (SVR) and sketch-based modeling provide more casual means for novices to customize 3D faces. Single-image 3D face reconstruction can be roughly divided into two streams, namely, photo-realistic human face reconstruction and caricature face reconstruction.…”

How Accurate Is Passive Stereo For 3d Face Reconstruction?

“…These methods link images to a 3D morphable face model [4,5,6,7] or directly to 3D geometry [8,9,10]. The most advanced of these methods learn animatable face details to enhance the quality of reconstructions [11,12,4].…”