Data-Driven Speech Animation Synthesis Focusing on Realistic Inside of the Mouth

Abstract

Speech animation synthesis is still a challenging topic in the field of computer graphics. Despite many challenges, representing detailed appearance of inner mouth such as nipping tongue's tip with teeth and tongue's back hasn't been achieved in the resulting animation. To solve this problem, we propose a method of data-driven speech animation synthesis especially when focusing on the inside of the mouth. First, we classify inner mouth into teeth labeling opening distance of the teeth and a tongue according to phoneme information. We then insert them into existing speech animation based on opening distance of the teeth and phoneme information. Finally, we apply patch-based texture synthesis technique with a 2,213 images database created from 7 subjects to the resulting animation. By using the proposed method, we can automatically generate a speech animation with the realistic inner mouth from the existing speech animation created by previous methods.

Face2Face: Real-Time Face Capture and Reenactment of RGB Videos

“…Berthouzoz et al [2] find a flexible number of in-between frames for a video sequence using shortest path search on a graph that encodes frame similarity. Kawai et al [18] re-synthesize the inner mouth for a given frontal 2D animation using a tooth and tongue image database; they are limited to frontal poses, and do not produce as realistic renderings as ours under general head motion.…”

Warp-guided GANs for single-photo facial animation

“…The work of [Kuster et al 2012] and [Ganin et al 2016] show their interests in manipulating the gaze of a single image. Other works, such as [Garrido et al 2015], focus on transferring lip motion to an existing target video and [Blanz et al 2003;Kawai et al 2013Kawai et al , 2014 focus on the realism of the mouth region. Facial manipulation techniques have also been introduced for purposes of data augmentation [Masi et al 2016], magnifying (or suppressing) expressions [Yang et al 2012], removing large-scale motion [Bai et al 2013], or face frontalization [Hassner et al 2015], where local edits are commonly performed without signiicant changes in facial expressions such as those in our cases.…”

Text-driven Speech Animation with Emotion Control

“…However, they mainly focused on accurate pronunciation for learning language and did not produce animations from the viseme label. To create more realistic speech animation, Kawai et al examined out the movement of the teeth and tongue along with the movement of the lips [10]. They focused on the tip of the tongue between the teeth or the back of the tongue, but this type of approach requires sufficient data collected from the subject in advance to produce animation results.…”

“…It should be able to express the shape of the lips that is precisely synchronized with the speaking voice. Numerous studies have presented ways to create visual speech animation with the speech track [1,2,3,4,5,6,7,8,9,10,11] while other approaches have focused on simulating facial movements from a set of physical properties [12,13,14,15] or synthesizing emotional expressions from given facial models [16,17,18,19,20]. For more natural and realistic facial animation, an explicit solution is needed to combine the lip movements and facial expressions into one animation sequence.…”