AU-Aware 3D Face Reconstruction through Personalized AU-Specific Blendshape Learning

Kuang, Chenyi; Cui, Zijun; Kephart, Jeffrey O.; Ji, Qiang

doi:10.1007/978-3-031-19778-9_1

“…Recently, various new loss functions and architectures have been introduced to address the limitations of existing methods with respect to reconstruction accuracy of the rich and detailed facial expressions [12,13,46,47]. In particular, the method of capturing emotions and reconstructing them into 3D faces demonstrates notable e cacy [12].…”

Section: Introductionmentioning

confidence: 99%

“…It is observed that that within the existing 3D face reconstruction process, there is commendable pro ciency in handling emotions, while the performance in encoding AUs is comparatively modest [48]. There exist a number of studies that have emphasized the importance of utilizing AUs in the process of 3D face reconstruction [46,47]. However, they do not explicitly consider the correlations between AUs occurring in the frame-based reconstruction process and require the use of AU labels during training, leading to a lack of guaranteed performance in in-the-wild scenarios.…”

Section: Introductionmentioning

confidence: 99%

Action Unit-Based 3D Face Reconstruction Using Transformers

Kim,

Wang,

Lee

2024

Preprint

0

View full text Add to dashboard Cite

The reconstruction of 3D face shapes and expressions from single 2D images remains unconquered due to the lack of detailed modeling of human facial movements such as the correlation between the different parts of faces. Facial action units (AUs), which represent detailed taxonomy of the human facial movements based on observation of activation of muscles or muscle groups, can be used to model various facial expression types. We present a novel 3D face reconstruction framework called AU feature-based 3D FAce Reconstruction using Transformer (AUFART) that can generate a 3D face model that is responsive to AU activation given a single monocular 2D image to capture expressions. AUFART leverages AU-specific features as well as facial global features to achieve accurate 3D reconstruction of facial expressions using transformers. We also introduce a loss function which is to force the learning toward the minimal discrepancy in AU activations between the input and rendered reconstruction. The proposed framework achieves an average F1 score of 0.39, outperforming state-of-the-art methods.

show abstract

“…Recently, various new loss functions and architectures have been introduced to address the limitations of existing methods with respect to reconstruction accuracy of the rich and detailed facial expressions [12,13,46,47]. In particular, the method of capturing emotions and reconstructing them into 3D faces demonstrates notable efficacy [12].…”

Section: Introductionmentioning

confidence: 99%

“…It is observed that that within the existing 3D face reconstruction process, there is commendable proficiency in handling emotions, while the performance in encoding AUs is comparatively modest [48]. There exist a number of studies that have emphasized the importance of utilizing AUs in the process of 3D face reconstruction [46,47]. However, they do not explicitly consider the correlations between AUs occurring in the frame-based reconstruction process and require the use of AU labels during training, leading to a lack of guaranteed performance in in-the-wild scenarios.…”

Section: Introductionmentioning

confidence: 99%

Action Unit-Based 3D Face Reconstruction Using Transformers

Kim,

Wang,

Lee

2024

Preprint

0

View full text Add to dashboard Cite

The reconstruction of 3D face shapes and expressions from single 2D images remains unconquered due to the lack of detailed modeling of human facial movements such as the correlation between the different parts of faces. Facial action units (AUs), which represent detailed taxonomy of the human facial movements based on observation of activation of muscles or muscle groups, can be used to model various facial expression types. We present a novel 3D face reconstruction framework called AU feature-based 3D FAce Reconstruction using Transformer (AUFART) that can generate a 3D face model that is responsive to AU activation given a single monocular 2D image to capture expressions. AUFART leverages AU-specific features as well as facial global features to achieve accurate 3D reconstruction of facial expressions using transformers. We also introduce a loss function which is to force the learning toward the minimal discrepancy in AU activations between the input and rendered reconstruction. The proposed framework achieves an average F1 score of 0.39, outperforming state-of-the-art methods.

show abstract