Audio2Face: Generating Speech/Face Animation from Single Audio with Attention-Based Bidirectional LSTM Networks

Abstract

We propose an end to end deep learning approach for generating real-time facial animation from just audio. Specifically, our deep architecture employs deep bidirectional long shortterm memory network and attention mechanism to discover the latent representations of time-varying contextual information within the speech and recognize the significance of different information contributed to certain face status. Therefore, our model is able to drive different levels of facial movements at inference and automatically keep up with the corresponding pitch and latent speaking style in the input audio, with no assumption or further human intervention. Evaluation results show that our method could not only generate accurate lip movements from audio, but also successfully regress the speaker's time-varying facial movements.

Beyond deep fakes: Conceptual framework, applications, and research agenda for neural rendering of realistic digital faces

“…Technologies have recently been demonstrated such as NVIDIA's Audio2Face: to produce audio-driven AI-based facial characters [18]. This allows for a synthesized voice in real-time to drive a realistic facial animation system or a stylised Animojis system.…”

“…In other words, it is not controlled by a real video, but by an external device. This is the most experimental form of face control as purely voice-based solutions are only just emerging from research labs [18]. This form of face control will be necessary for the creation of virtual assistants as shown in type 4 in section 3.…”

Learning Pose-Adaptive Lip Sync with Cascaded Temporal Convolutional Network

“…Recent advances in deep learning first boost lip-sync research at the audio-to-mouth stage. RNN-based architectures [3,6] facilitates the learning of the sequential mapping from audio signals to mouth movements. In the rendering stage, ObamaNet [2] is a representative work that demonstrates the power of neural networks [7] in synthesizing the photorealistic appearance.…”

“…In the rendering stage, ObamaNet [2] is a representative work that demonstrates the power of neural networks [7] in synthesizing the photorealistic appearance. These methods [8,6] provide a fully-trainable solution for the classical two-stage lip-sync scheme, which significantly improves both the lip-sync accuracy and processing efficiency.…”

“…The main options are 2D Facial Landmarks (2DFLs) and 3D Morphable Models (3DMMs). 3DMMs have the strength of controllable parameters and free rotation [8,6]. However, they usually [10,11] require landmarks for registration, and their parametric nature leads to a weakening of personal talking style.…”