FaceFormer: Speech-Driven 3D Facial Animation with Transformers

3D head-talk: speech synthesis 3D head movement face animation

“…Evaluation of mouth synchronization. We first followed the lip synchronization metrics used in [18] and [22] to assess the quality of lip movements. By defining the maximum L2 error for all lip vertices as the lip error per frame.…”

“…In this way audio and facial motion models can be aligned by cross-modal multiheaded attention with bias [18].…”

“…Next we apply RPE to f 𝑡 to provide temporal information periodically to the encoded vector representation. Next, similar to [18], a causal multi-headed attention mechanism with linear bias is introduced. First linearly the sequence of temporally encoded facial motion representations F ̂t=(f ̂1,...,f ̂t) are projected to the query Q F ̂ and key K F ̂ of dimension d k , and the value V F ̂ of dimension d v .…”

“…To learn the dependencies between each frame in past facial motion sequences, causality (in Figure 1) was ensured by performing scaled dot product notation and adding B^F ̂ linear time deviations [27] to compute a weighted contextual representation and improve the model's ability to model longer sequences. On this basis, the obtained representations are made cross-attentive, again introducing the cross-modal multiheaded attention (in Figure 1) with bias crossover proposed by [18], allowing the audio and motion representations to be aligned. Attention scores are obtained for each token (token) of each facial movement, and these attention scores are further weighted by the similarity previously obtained after biased causal attention.…”

“…MeshTalk [17] considers long-term temporal context by modeling audio sequences, but its conversion of audio to mel acoustic spectrogram features to train the model cannot synthesize accurate lip motion. Fan et al [18], in order to address the long-term contextual background and the problem of sparse 3D audiovisual data, a transformer [19] based regression model was proposed. However, the above studies ignore the simultaneous representation of lip movements and other facial movements (e.g., head movements), which are inevitable during speech.…”