To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations

Abstract

Non verbal behaviours such as gestures, facial expressions, body posture, and para-linguistic cues have been shown to complement or clarify verbal messages. Hence to improve telepresence, in form of an avatar, it is important to model these behaviours, especially in dyadic interactions. Creating such personalized avatars not only requires to model intrapersonal dynamics between a avatar's speech and their body pose, but it also needs to model interpersonal dynamics with the interlocutor present in the conversation. In this paper, we introduce a neural architecture named Dyadic Residual-Attention Model (DRAM), which integrates intrapersonal (monadic) and interpersonal (dyadic) dynamics using selective attention to generate sequences of body pose conditioned on audio and body pose of the interlocutor and audio of the human operating the avatar. We evaluate our proposed model on dyadic conversational data consisting of pose and audio of both participants, confirming the importance of adaptive attention between monadic and dyadic dynamics when predicting avatar pose. We also conduct a user study to analyze judgments of human observers. Our results confirm that the generated body pose is more natural, models intrapersonal dynamics and interpersonal dynamics better than non-adaptive monadic/dyadic models.

Gesticulator: A framework for semantically-aware speech-driven gesture generation

“…To obtain semantic information for the speech, we first transcribed the audio recordings using Google Cloud automatic speech recognition (ASR), followed by thorough manual review to correct recognition errors and add punctuation for both the training and test parts of the dataset. The same data was used by the GENEA 2020 gesture generation challenge 1 and has been made publicly available in the original dataset repository 2 .…”

“…Those approaches are constrained by the discrete set of gestures they can produce. Alongside recent advances in deep learning, data-driven approaches have increasingly gained interest for gesture generation [1,27,48]. While early work has considered gesture generation as a classification task which aims to deduce a specified gesture class [9,37], more recent work has considered it as a regression task which aims to produce continuous motion [2,48].…”

Let's Face It

“…Our stimulus-generation method allowed manipulating speech articulation independently from the facial gestures, allowing for varying the facial gestures while controlling for the effect of speech context. We find that: (1) Evaluators can distinguish mimicry segments from mismatched segments (from the same interaction but another point in time) and find mimicry segments more appropriate. This also validates that our feature extraction and stimulus generation methods are appropriate for non-verbal behavior.…”

“…Our problem formulation is largely inspired by a recent method to model conversational dynamics for gesture generation [1]. Like in that work, we also model avatar behavior based on both the avatar's own speech and the speech and motion of the interlocutor.…”

“…As modeling conversational dynamics is difficult to achieve, most non-verbal behavior generation methods only use speech and/or semantic content produced by the agent as inputs to the system [27,32,45]. Recently a few systems have been introduced that use non-verbal behaviors from the interlocutor to control non-verbal output from the system [1,17,20,25]. We continue this line of work and present a probabilistic system, based on normalizing flows, for generating facial gestures in dyadic settings.…”