Emotional facial expression transfer from a single image via generative adversarial nets

Abstract

Facial expression transfer from a single image is a challenging task and has drawn sustained attention in the fields of computer vision and computer graphics. Recently, generative adversarial nets (GANs) have provided a new approach to facial expression transfer from a single image toward target facial expressions.However, it is still difficult to obtain a sequence of smoothly changed facial expressions. We present a novel GAN-based method for generating emotional facial expression animations given a single image and several facial landmarks for the in-between stages. In particular, landmarks of other subjects are incorporated into a GAN model to control the generated facial expression from a latent space. With the trained model, high-quality face images and a smoothly changed facial expression sequence can be effectively obtained, which are showed qualitatively and quantitatively in our experiments on the Multi-PIE and CK+ data sets. KEYWORDSdynamic expression sequence, facial expression transfer, generative adversarial nets *These authors contributed equally to this work.

Generative Adversarial Networks in Human Emotion Synthesis: A Review

“…Other alternative models that benefit from a different loss metric are GAN based on Category Information (CIGAN) [57], hinge loss [47], least-square GAN [58], and f-divergence FIGURE 2: Block diagram of LAPGAN model [41]; G k : k-th Generator, D k : k-th Discriminator, X r1 : real sample, X rk : k-th real residual, X gk : generated sample, O 1 : Output of binary classification to real/fake. (a) BiGAN/ALI [61] (b) VAEGAN [62] FIGURE 4: General Structure of GAN models varying by generator; Z: input noise, G:Generator, D:Discriminator, x r : real sample, x g : generated sample, O 1 : Output of binary classification to real/fake, G e : incorporated Generator-encoder, G d : incorporated Generator-decoder, Z r : latent vector that encodes the input for G e , Z g : latent vector that encodes the input for G d .…”

“…In 2018, the G2GAN [98] was extended by Qiao et al [62]. The authors derived a model based on VAEGANs to…”

The Many Variations of Emotion

“…More recently, another approach [34] is using fiducial points to geometrically control the face animation while Tulyakov et al [36] is learning to directly generate sequences of images, based on a "content and motion" approach. Quia et al [27] use facial landmarks to improve the animation smoothness of a changing emotion. Kim et al [15] enable to generate video face animation using another portrait video as an example.…”

“…For generating artificial representation of emotions, first works from computer graphics community focus on the animation of the faces with model-based approaches [15], [33], [41]. More recently deep learning approaches and especially Generative Adversarial Networks [5], [7], [11], [27], [36] have been proposed, often borrowing ideas from the computer graphics community [26], [34] but also aiming to learn these facial expressions from diverse datasets and representations. Nevertheless, these approaches are generally trained on small corpus with pronounced emotions [19], [20], [43], meaning that the space of possible generated expressions is limited.…”

All-In-One: Facial Expression Transfer, Editing and Recognition Using A Single Network

“…Pumarola et al [23] and Shao et al [28] exploited Facial Action Units (AU), and the expression synthesis process is guided by the learned AU features. Similarly, in [30] and [25], facial landmarks are used to produce synthesized expression images.…”