Audio-to-Visual Speech Conversion Using Deep Neural Networks

Abstract

We study the problem of mapping from acoustic to visual speech with the goal of generating accurate, perceptually natural speech animation automatically from an audio speech signal. We present a sliding window deep neural network that learns a mapping from a window of acoustic features to a window of visual features from a large audio-visual speech dataset. Overlapping visual predictions are averaged to generate continuous, smoothly varying speech animation. We outperform a baseline HMM inversion approach in both objective and subjective evaluations and perform a thorough analysis of our results. Index Terms: Audio-to-visual conversion, automatic speech animation, sliding window deep neural networks.

Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks

“…DNN-based models directly learn how to predict the movements from speech features. Taylor et al [3] proposed a fully connected feedforward neural network for audio to visual conversion. Their network gets the speech features over a specified contextual window, predicting current and future orofacial movements.…”

“…The integration between these factors in the orofacial area is complex [1,2]. Most of previous studies on lip movement synthesis have relied on the recordings from one subject in order to avoid speaker variations [3,4,5]. Since multimodal emotional corpora usually include multiple speakers with limited data per subject [6], it is important that the models can effectively capture speaker variability.…”

“…The resulting lip motion sequences capture the temporal coupling between speech and lip movements, creating realistic sequences, which convey the underlying lexical content. We compare the CSG model with three baselines proposed in previous studies [3,13,14], which use conventional nonadversarial methods. The experimental evaluations with objective and subjective metrics demonstrate that the proposed CSG model achieves better performance than these methods.…”

Speaker-Independent Speech-Driven Visual Speech Synthesis using Domain-Adapted Acoustic Models

“…via phonemic transcription. For direct approaches, the conversion function typically involves some form of regression [16,24,26,27] or indexing a codebook of visual features using the corresponding features extracted from the acoustic speech [3,13]. For indirect approaches, the mapping function involves concatenation or interpolation of pre-existing data [5,7,9,21,29] or using a generative model [2,10,17].…”

Research Paper on Expressive Facial Animation Synthesis by Learning Speech Co-articulation and Expression Spaces

“…The human face has very complex muscular structure and bone structure. The human face contains different complex muscles and expression by a human is normally made by the stretching and contracting of a facial muscle or by relaxing or stressing of facial muscles [1] . For the current project, video of a face was taken that has hundreds of facial marks with the help of optical motion capture system.…”