Towards the generation of synchronized and believable non-verbal facial behaviors of a talking virtual agent

Abstract

This paper introduces a new model to generate rhythmically relevant non-verbal facial behaviors for virtual agents while they speak. The model demonstrates perceived performance comparable to behaviors directly extracted from the data and replayed on a virtual agent, in terms of synchronization with speech and believability. Interestingly, we found that training the model with two different sets of data, instead of one, did not necessarily improve its performance. The expressiveness of the people in the dataset and the shooting conditions are key elements. We also show that employing an adversarial model, in which fabricated fake examples are introduced during the training phase, increases the perception of synchronization with speech. A collection of videos demonstrating the results and code can be accessed at: https://github.com/aldelb/non_verbal_facial_animation. CCS CONCEPTS• Computing methodologies → Neural networks; Animation.

A Survey on Realistic Virtual Human Animations: Definitions, Features and Evaluations

“…It relies on the frequency components of the movement's speed profile (i.e., changes of speed over time), represented using the Fourier magnitude spectrum. Closely related to this motion invariant but sharing the data‐driven comparison, let us also mention the study of Delbosc et al [DOS*23] which aimed at generating synchronized and believable facial non‐verbal animations for conversational VH. Authors proposed to evaluate their resulting animation in comparison to ground‐truth data both using a distance metrics based on DTW but also comparing jerk results as a good indicator of motion naturalness.…”