Mesh-Tension Driven Expression-Based Wrinkles for Synthetic Faces

HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details

“…To train the network end-to-end, recent methods leverage the differentiable renderers [28,22,80,49], along with the photo loss, perceptual loss, and landmark loss [21,19,28,67,71] to optimize the network in a self-supervised manner. Different from these coarse shape Given a monocular image, we regress its shape and detail coefficients to synthesize a realistic 3D face, and leverage a differentiable renderer [28] to train the whole model end-to-end from synthetic [72,57] and real-world [40,52] images. (b).…”

“…We explicitly decouple the static and dynamic factors to synthesize realistic and animatable details. Given the shape and static coefficients, we regress the static and dynamic details through displacement bases and interpolate them into the final details through vertex tension [57].…”

“…Therefore, we propose HiFace to explicitly model the static and dynamic details for high-fidelity 3D face reconstruction. More specifically, for person-specific static detail, instead of directly predicting the displacement map that may increase the difficulty of detail prediction [24,19], we follow the spirit of 3DMMs to build a displacement basis from the captured facial scans with age diversity [57,72]. In this way, the model is trained to predict the coefficients of the displacement basis, and make the detail prediction easier.…”

“…For dynamic detail, since it is highly expressiondependent, directly modeling it with one displacement basis is quite difficult. Therefore, based on the fact that the expression can be interpolated by a compressed and a stretched expressions [57], we build two displacement bases for the compressed and stretched expressions from the captured scans respectively, and learn to regress the displacement coefficients with the ground-truth labels. Therefore, we can obtain the dynamic detail by linearly interpolating the compressed and stretched displacement maps, which are derived from the displacement bases and the predicted coefficients.…”

“…Since the final output consists of both the coarse shape and details, and is typically reconstructed for real-world images, we propose several novel loss functions to learn coarse shape and details simultaneously from both the synthetic and real-world datasets. For details, we leverage the ground-truth static and dynamic displacement maps of the synthetic dataset [72,57] as supervision. While for the coarse shape, we leverage the ground-truth vertex of the synthetic dataset as supervision.…”