CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior

Abstract

Speech-driven 3D facial animation has been widely studied, yet there is still a gap to achieving realism and vividness due to the highly ill-posed nature and scarcity of audiovisual data. Existing works typically formulate the crossmodal mapping into a regression task, which suffers from the regression-to-mean problem leading to over-smoothed facial motions. In this paper, we propose to cast speechdriven facial animation as a code query task in a finite proxy space of the learned codebook, which effectively promotes the vividness of the generated motions by reducing the cross-modal mapping uncertainty. The codebook is learned by self-reconstruction over real facial motions and thus embedded with realistic facial motion priors. Over the discrete motion space, a temporal autoregressive model is employed to sequentially synthesize facial motions from the input speech signal, which guarantees lip-sync as well as plausible facial expressions. We demonstrate that our approach outperforms current state-of-the-art methods both qualitatively and quantitatively. Also, a user study further justifies our superiority in perceptual quality. The source code will be publicly available.

Say Anything with Any Style

“…Among the methods for discrete representation learning, the VQ-VAE approach (Van Den Oord, Vinyals et al 2017) has gained significant popularity for quantizing latent features into a learned codebook. Ng et al (2022) and Xing et al (2023) their approaches, we explore a multi-task VQ-VAE to extract the speaking style with the help of a learned codebook.…”

Action Unit-Based 3D Face Reconstruction Using Transformers

“…3D face reconstruction from 2D images has received a tremendous amount of attention in computer vision and has made major progresses thanks to the highly accurate modeling capability of deep learning. 3D face reconstruction enables a wide range of applications such as speech-driven 3D facial animation, 3D avatar generation, virtual makeup, performance capture, virtual and augmented reality, and human-robot interaction [2][3][4][5][6][7].…”

Action Unit-Based 3D Face Reconstruction Using Transformers

“…3D face reconstruction from 2D images has received a tremendous amount of attention in computer vision and has made major progresses thanks to the highly accurate modeling capability of deep learning. 3D face reconstruction enables a wide range of applications such as speechdriven 3D facial animation, 3D avatar generation, virtual makeup, performance capture, virtual and augmented reality, and human-robot interaction [2][3][4][5][6][7].…”