3D-CariGAN: An End-to-End Solution to 3D Caricature Generation From Normal Face Photos

Abstract

Caricature is a type of artistic style of human faces that attracts considerable attention in the entertainment industry. So far a few 3D caricature generation methods exist and all of them require some caricature information (e.g., a caricature sketch or 2D caricature) as input. This kind of input, however, is difficult to provide by non-professional users. In this paper, we propose an end-to-end deep neural network model that generates high-quality 3D caricatures directly from a normal 2D face photo. The most challenging issue for our system is that the source domain of face photos (characterized by normal 2D faces) is significantly different from the target domain of 3D caricatures (characterized by 3D exaggerated face shapes and textures). To address this challenge, we:(1) build a large dataset of 5,343 3D caricature meshes and use it to establish a PCA model in the 3D caricature shape space; (2) reconstruct a normal full 3D head from the input face photo and use its PCA representation in the 3D caricature shape space to establish correspondences between the input photo and 3D caricature shape; and (3) propose a novel character loss and a novel caricature loss based on previous psychological studies on caricatures. Experiments including a novel two-level user study show that our system can generate high-quality 3D caricatures directly from normal face photos.

Improving Face Recognition Using Artistic Interpretations of Prominent Features: Leveraging Caricatures in Modern Surveillance Systems

“…Despite these limitations, recent work generates a caricature from veridical images using GANs [41,[50][51][52], but does not try to understand or utilize caricatures to improve verification or recognition. Some work attempts to exploit caricatures to improve verification.…”

Tag-based annotation creates better avatars

“…Other methods are guided by position and landmarks, extracting human facial features used to deform textures and meshes [24][25][26][27] . Recently a conditional GAN has been applied in the generation process 28,29 . However, these methods all utilized parameters without semantic meaning, making them inapplicable to avatar systems designed to provide user level customization of asset choice.…”

Exemplar-Based 3D Portrait Stylization

“…Recently, some deep learning-based interactive systems [35], [36] are proposed for modeling 3D caricature faces by interactively drawing sketches. The approach [37] builds a 3D parametric model for caricature face reconstruction targeting the problem of limited extrapolation capability of the existing parametric models based on normal faces, and support a certain level of user control on the reconstructed faces. Our work addresses a new challenge of automatically transferring geometry and texture styles from an arbitrary portrait art image to a 3D face model, which is not limited in the caricature domain.…”