Realistic Speech-Driven Facial Animation with GANs

Abstract

Speech-driven facial animation is the process that automatically synthesizes talking characters based on speech signals. The majority of work in this domain creates a mapping from audio features to visual features. This approach often requires post-processing using computer graphics techniques to produce realistic albeit subject dependent results. We present an end-to-end system that generates videos of a talking head, using only a still image of a person and an audio clip containing speech, without relying on handcrafted intermediate features. Our method generates videos which have (a) lip movements that are in sync with the audio and (b) natural facial expressions such as blinks and eyebrow movements. Our temporal GAN uses 3 discriminators focused on achieving detailed frames, audio-visual synchronization and realistic expressions. We quantify the contribution of each component in our model using an ablation study and we provide insights into the latent representation of the model. The generated videos are evaluated based on sharpness, reconstruction quality, lip-reading accuracy, synchronization as well as their ability to generate natural blinks.

Visually Guided Self Supervised Learning of Speech Representations

“…The proposed architecture is shown in Fig. 1 and is based on our prior work on speech-driven facial animation [17]. The model is a temporal encoder-decoder which takes a still image (frame from a 25 fps video) and an audio singal as input.…”

“…multi-task speech representations by leveraging the visual modality (inspired by our prior work [17]). Specifically, we make the following research contributions: (i) We animate a still image to generate speech video by conditioning on the corresponding audio.…”

Speech-Driven Facial Animation Using Polynomial Fusion of Features

“…We can perform further parameter sharing by assuming that the mode-2 and mode-3 factor matrices are equivalent to the matrices describing the row spaces, i.e. : B (2)…”

“…We ran experiments on the above datasets using the following methodologies for the polynomial fusion layer: (3), (4), (5), (6b) (PF-CMF-SR) We set a = 256, d = 128, n = 10 and trained on video sequences of 3 seconds with frame size 128 × 96 as per [2]. For all models, c = a + d + n = 394, implying m = 384, given that c = m + n by construction.…”

“…CGI traditionally employs face capture methods for facial synthesis [2], which are costly and require manual labor. To alleviate this, recent research has focused on automatic generation of video with machine learning [1,2,3,4,5]. MoCo-GAN [4] models motion and content as separate latent spaces, which are learned in an unsupervised way using both image and video discriminators.…”