Text-driven Mouth Animation for Human Computer Interaction With Personal Assistant

Abstract

Personal assistants are becoming more pervasive in our envi-ronments but still do not provide natural interactions. Their lack of realism in term of expressiveness and their lack of visual feedback can create frustrating experiences and make users lose patience. In this sense, we propose an end-to-end trainable neural architecture for text-driven 3D mouth animations. Previous works showed such architectures provide better realism and could open the door for integrated affective Human Computer Interface (HCI). Our study shows that such visual feedback improves users’ comfort for 78%of the candidates significantly while slightly improving their time perception.

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person

“…Similar to some recent text-driven talking head generation methods [5], [6], [11], our method uses TTS to synthesize the audio track. The goal of a TTS system is to synthesize human-like speech from a natural language text input.…”

“…Therefore, facial landmarks can be used to represent the facial-related characteristics, e.g., face shapes, head poses, and mouth shapes, and it is easy to build mapping relation between facial landmarks and the facial expression in a photo. Recently, the facial landmarks have been popular intermediate representations to bridge the gap between the raw audio signal and photo-realistic videos in recent research [42], [5], [43], [10], [44]. However, these methods suffer from several limitations, such as only can be used for the person that used for the training data [42] and can not be used for arbitrary persons, depending on reference videos to provide pose information [42], [44], or no head movement is predicted and can only present static head pose [10].…”

“…2) Database for face encoder: The speaker embedding extractor which works as the teacher to train the face encoder is trained with the database of Aidatatang-200zh 5 . This is a Chinese Mandarin speech corpus that contains 200 hours of speech data from 600 speakers.…”

“…Compared to the audio-driven and video-driven methods, the text-driven method is more flexible, as it allows users to create any new content because it is not dependent on an existing corpus or on source videos. Although there are several textdriven methods that directly use textual phonetic labels to predict the visual speech [4], most of the recent text-driven methods [1], [5], [6] decompose the text-to-video process into separate text-to-speech (TTS) and speech-to-video processes with a TTS module, i.e., 1) synthesize speech with text as input using the TTS module and 2) perform the audiodriven talking head generation with synthesized speech as input. As a TTS module is indispensable both in the phonetic label-to-video method and text-to-speech-to-video method for building a talking-head video with synchronized audio, in this work, we follow the latter strategy, which allows us to use the intermediate representation of synthesized speech, e.g., spectrograms, to generate the synchronized video.…”