VOCAL: Vowel and Consonant Layering for Expressive Animator-Centric Singing Animation

Pan, Yuwei; Landreth, Chris; Fiume, Eugene; Singh, Karan

doi:10.1145/3550469.3555408

Cited by 4 publications

(2 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Audiovisual speech animation can be categorized into procedural, performance-capture, data-driven, and deep-learning techniques [24,40]. Procedural speech animation uses visemes, the oral shape at the apex of a given phoneme, to solve overlaps between successive visemes, curve shapes that define attach, apex, sustain, and decay of viseme, and mapping a given phoneme to a viseme [33].…”

Section: Audio Driven Facial Animationmentioning

confidence: 99%

Expressive Talking Avatars

Pan,

Tan,

Cheng

et al. 2024

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Stylized avatars are common virtual representations used in VR to support interaction and communication between remote collaborators. However, explicit expressions are notoriously difficult to create, mainly because most current methods rely on geometric markers and features modeled for human faces, not stylized avatar faces. To cope with the challenge of emotional and expressive generating talking avatars, we build the Emotional Talking Avatar Dataset which is a talking-face video corpus featuring 6 different stylized characters talking with 7 different emotions. Together with the dataset, we also release an emotional talking avatar generation method which enables the manipulation of emotion. We validated the effectiveness of our dataset and our method in generating audio based puppetry examples, including comparisons to state-of-the-art techniques and a user study. Finally, various applications of this method are discussed in the context of animating avatars in VR.

show abstract

Section: Audio Driven Facial Animationmentioning

confidence: 99%

Expressive Talking Avatars

Pan,

Tan,

Cheng

et al. 2024

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

show abstract

“…It is interesting to note that music-driven singing face synthesis remains a rarely studied, open problem. Song2Face [53] and VOCAL [54] appear to be the only two designed methods for singing up to now. However, they take as input a plain human singing voice, and only works well in the absence of backing music.…”

Section: Music-driven Animationmentioning

confidence: 99%

MusicFace: Music-driven expressive singing face synthesis

Liu,

Deng,

et al. 2023

Comp. Visual Media

View full text Add to dashboard Cite

It remains an interesting and challenging problem to synthesize a vivid and realistic singing face driven by music. In this paper, we present a method for this task with natural motions for the lips, facial expression, head pose, and eyes. Due to the coupling of mixed information for the human voice and backing music in common music audio signals, we design a decouple-and-fuse strategy to tackle the challenge. We first decompose the input music audio into a human voice stream and a backing music stream. Due to the implicit and complicated correlation between the two-stream input signals and the dynamics of the facial expressions, head motions, and eye states, we model their relationship with an attention scheme, where the effects of the two streams are fused seamlessly. Furthermore, to improve the expressivenes of the generated results, we decompose head movement generation in terms of speed and direction, and decompose eye state generation into short-term blinking and long-term eye closing, modeling them separately. We have also built a novel dataset, SingingFace, to support training and evaluation of models for this task, including future work on this topic. Extensive experiments and a user study show that our proposed method is capable of synthesizing vivid singing faces, qualitatively and quantitatively better than the prior state-of-the-art.

show abstract