Local control editing paradigms for part‐based 3D face morphable models

Abstract

We propose an approach to construct realistic 3D facial morphable models (3DMM) that allows an intuitive facial attribute editing workflow. Current face modeling methods using 3DMM suffer from a lack of local control. We thus create a 3DMM by combining local part-based 3DMM for the eyes, nose, mouth, ears, and facial mask regions. Our local principal component analysis (PCA)-based approach uses a novel method to select the best eigenvectors from the local 3DMM to ensure that the combined 3DMM is expressive, while allowing accurate reconstruction. We provide different editing paradigms, all designed from the analysis of the data set. Some use anthropometric measurements from the literature and others allow the user to control the dominant modes of variation extracted from the data set. Our part-based 3DMM is compact, yet accurate, and compared to other 3DMM methods, it provides a new trade-off between local and global control. We tested our approach on a data set of 135 scans used to derive the 3DMM, plus 19 scans that served for validation.The results show that our part-based 3DMM approach has excellent generative properties and allows the user intuitive local control.

Face Editing Using Part‐Based Optimization of the Latent Space

“…As a result, it ensures that the deformation only appears locally. Therefore, the method of Ghafourzadeh et al [GFR*21] performs better regarding reconstruction quality and keeping the changes local. Nonetheless, in Figure 10, we observe that our model deforms the face more naturally and meaningfully while keeping the changes comparably local.…”

“…Also, it segments the face, similar to our method. The method of [GFR*21] deforms each part in isolation and uses an additional smooth blending step to merge the generated part into the face. As a result, it ensures that the deformation only appears locally.…”

“…Classical 3D morphable face model (3DMM) methods as well as many recent ones [CWZ*14, BRZ*16, LBB*17, BRP*18, PWP*19, PVO*20, GRF*20, EST*20] use PCA to sample the distribution of the face models. The popularity of PCA‐based methods is due to their simplicity and efficiency.…”

“…Local control is less important when the application is global face reconstruction, but it is a critical limitation when the application is face editing. To address this limitation, Ghafourzadeh et al [GRF*20] propose a method that further decomposes the face into semantic parts allowing independent generation for each part followed by an ARAP‐inspired reassembling of the parts into one coherent face mesh. Vertex‐based editing was added to this part‐based method [GFR*21].…”

“…To address this limitation, Ghafourzadeh et al [GRF*20] propose a method that further decomposes the face into semantic parts allowing independent generation for each part followed by an ARAP‐inspired reassembling of the parts into one coherent face mesh. Vertex‐based editing was added to this part‐based method [GFR*21]. Similarly to our approach, the user edits the face by moving mesh vertices.…”