Style‐Controllable Speech‐Driven Gesture Synthesis Using Normalising Flows

Abstract

Automatic synthesis of realistic gestures promises to transform the fields of animation, avatars and communicative agents. In off‐line applications, novel tools can alter the role of an animator to that of a director, who provides only high‐level input for the desired animation; a learned network then translates these instructions into an appropriate sequence of body poses. In interactive scenarios, systems for generating natural animations on the fly are key to achieving believable and relatable characters. In this paper we address some of the core issues towards these ends. By adapting a deep learning‐based motion synthesis method called MoGlow, we propose a new generative model for generating state‐of‐the‐art realistic speech‐driven gesticulation. Owing to the probabilistic nature of the approach, our model can produce a battery of different, yet plausible, gestures given the same input speech signal. Just like humans, this gives a rich natural variation of motion. We additionally demonstrate the ability to exert directorial control over the output style, such as gesture level, speed, symmetry and spacial extent. Such control can be leveraged to convey a desired character personality or mood. We achieve all this without any manual annotation of the data. User studies evaluating upper‐body gesticulation confirm that the generated motions are natural and well match the input speech. Our method scores above all prior systems and baselines on these measures, and comes close to the ratings of the original recorded motions. We furthermore find that we can accurately control gesticulation styles without unnecessarily compromising perceived naturalness. Finally, we also demonstrate an application of the same method to full‐body gesticulation, including the synthesis of stepping motion and stance.

Gesticulator: A framework for semantically-aware speech-driven gesture generation

“…Future work also involves making the model stochastic (as in [2]), using larger datasets (such as [29]) and further improving the semantic coherence of the gestures, for instance by treating different gesture types separately.…”

“…The recurrent connections used in several models [13,19,48] can also act as a pose memory that may help the model to produce smooth output motion. Autoregressive motion models have recently demonstrated promising results in probabilistic audio-driven gesture generation [2]. In this paper, we similarly investigate autoregressive connections for improving motion quality, which explicitly provide the most recent poses as input to the model when generating the next pose.…”

“…Alongside recent advances in deep learning, data-driven approaches have increasingly gained interest for gesture generation [1,27,48]. While early work has considered gesture generation as a classification task which aims to deduce a specified gesture class [9,37], more recent work has considered it as a regression task which aims to produce continuous motion [2,48]. We focus on the latter task: continuous gesture generation.…”

Understanding the Predictability of Gesture Parameters from Speech and their Perceptual Importance

“…In recent work, recurrent neural networks have proven popular; a classic training loss has been employed for English [12,21] and Japanese speech-to-gesture generation [18,20]. To combat the problem of mean pose regression in a standard training paradigm, an adversarial training paradigm has been proposed in [14] (similarly for a convolutional network setup in [15]), and recently, probabilistic generative modelling has shown promise [1]. However, due to the highly indeterministic input-to-output relation, modelling plausible gestures remains a difficult problem.…”

“…(3.1) path length (3.2) major axis length (4) arm swivel (5) hand opening Velocity and initial acceleration both describe the kinematics of the gesture, represented by the maximum stroke velocity (1), and by the mean acceleration to the first major velocity peak (2). Velocity captures a character's tempo and relates to the amount of energy they are using.…”