Expressive Talking Head Generation with Granular Audio-Visual Control

Liang, Borong; Yan, Pingkun; Guo, Zhizhi; Zhou, Hang; Hong, Zhibin; Han, Xiaoguang; Han, Junyu; Liu, Jingtuo; Ding, Errui; Wang, Jingdong

doi:10.1109/cvpr52688.2022.00338

Cited by 81 publications

(27 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a branch, speechdriven facial animation is to reenact a person in sync with input speech sequences. While extensive literature in this field works on 2D talking heads [1,[7][8][9]11,23,28,29,38,39,48,52,62,64,68,69], we focus on facial animation on 3D models in this work, which can be roughly categorized into linguistics-based and learning-based methods.…”

Section: Speech-driven 3d Facial Animationmentioning

confidence: 99%

CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior

Xing¹,

Mao²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Speech-driven 3D facial animation has been widely studied, yet there is still a gap to achieving realism and vividness due to the highly ill-posed nature and scarcity of audiovisual data. Existing works typically formulate the crossmodal mapping into a regression task, which suffers from the regression-to-mean problem leading to over-smoothed facial motions. In this paper, we propose to cast speechdriven facial animation as a code query task in a finite proxy space of the learned codebook, which effectively promotes the vividness of the generated motions by reducing the cross-modal mapping uncertainty. The codebook is learned by self-reconstruction over real facial motions and thus embedded with realistic facial motion priors. Over the discrete motion space, a temporal autoregressive model is employed to sequentially synthesize facial motions from the input speech signal, which guarantees lip-sync as well as plausible facial expressions. We demonstrate that our approach outperforms current state-of-the-art methods both qualitatively and quantitatively. Also, a user study further justifies our superiority in perceptual quality. The source code will be publicly available.

show abstract

Section: Speech-driven 3d Facial Animationmentioning

confidence: 99%

CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior

Xing¹,

Mao²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Audio-driven talking head generation [13,19,38,14] is another popular direction on this topic, as audio sequences do not contain information of the face identity, and is relatively easier to disentangle the motion information from the input audio. Liang et al [16] explicitly divide the driving audio into granular parts through delicate priors to control the lip shape, face pose, and facial expression.…”

Section: Related Workmentioning

confidence: 99%

Depth-Aware Generative Adversarial Network for Talking Head Video Generation

Hong

Zhang²,

Shen³

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Predominant techniques on talking head generation largely depend on 2D information, including facial appearances and motions from input face images. Nevertheless, dense 3D facial geometry, such as pixel-wise depth, plays a critical role in constructing accurate 3D facial structures and suppressing complex background noises for generation. However, dense 3D annotations for facial videos is prohibitively costly to obtain. In this work, firstly, we present a novel self-supervised method for learning dense 3D facial geometry (i.e., depth) from face videos, without requiring camera parameters and 3D geometry annotations in training. We further propose a strategy to learn pixel-level uncertainties to perceive more reliable rigid-motion pixels for geometry learning. Secondly, we design an effective geometry-guided facial keypoint estimation module, providing accurate keypoints for generating motion fields. Lastly, we develop a 3D-aware cross-modal (i.e., appearance and depth) attention mechanism, which can be applied to each generation layer, to capture facial geometries in a coarse-to-fine manner. Extensive experiments are conducted on three challenging benchmarks (i.e., VoxCeleb1, VoxCeleb2, and HDTF). The results demonstrate that our proposed framework can generate highly realistic-looking reenacted talking videos, with new state-of-the-art performances established on these benchmarks. The codes and trained models are publicly available on the GitHub project page.

show abstract

“…The W-GAN learns the distribution of facial expression dynamics of different classes, from which new facial expression motions are synthesized and transformed to videos by Texture-GAN. Other works have investigated guiding facial expression generation by speech audio data, such as (Chen et al, 2020;Guo et al, 2021;Wang et al, 2022;Liang et al, 2022), or by a combination of audio and facial landmark information, like (Wang et al, 2021;Wu et al, 2021;Sinha et al, 2022) . All the methods mentioned before are methods that generate a single frame at a time-step, which lowers the dependency between the video frames causing the lack of spatio-temporal consistency.…”

Section: Related Workmentioning

confidence: 99%