Speech-driven facial animation with spectral gathering and temporal attention

Voice2Face: Audio‐driven Facial and Tongue Rig Animations with cVAEs

“…Closer to our approach, a number of works produce 3D animations directly from speech [KAL*17; PWP18; TPL*20; RZW*21; CWWZ22]. Using formants as sound representation, Karras et al [KAL*17] achieves impressive results from less than 4 minutes of training data.…”

“…More recently, Chai et al [CWWZ22] gathers information along the frequency dimension of a speech window with a stack of convolutions, but uses self‐attention layers to collect information along the time dimension. Similar to Cudeiro et al [CBL*19], their model takes speaker identity as auxiliary input and is thus able to explicitly model different speaking styles.…”

“…Our approach is designed to produce lip sync suitable for bulk animations for a wide range of characters, as a lighter weight option to performance captures. Unlike [CBL*19; KAL*17; CWWZ22], we aim to produce generic animations, without speaker‐dependent characteristics. In contrast to e.g.…”

“…Recently, a number of methods have been proposed using Deep Learning (DL) to train speech‐driven models directly on performance capture‐based data [KAL*17; CBL*19; RZW*21; CWWZ22], generating animations with the qualities attributed to performance captures but requiring only recorded speech as input. While these works excel at e.g.…”

3D head-talk: speech synthesis 3D head movement face animation

“…There are several methods [10][11][12] to obtain 3D facial parameter representations from 2D monocular videos, but the quality of the synthesized 3D data receives limitations in the accuracy of 3D reconstruction techniques and 3D reconstruction techniques cannot realize subtle changes in 3D based on 2D videos, so this may lead to unreliable results. In works that generate 3D facial animations based on 3D meshes [13][14][15], they delay the speech input to short audio windows, which may lead to pauses in lip movements with speech changes, which further may affect the realistic facial changes.…”