Analyzing Input and Output Representations for Speech-Driven Gesture Generation

Abstract

This paper presents a novel framework for automatic speech-driven gesture generation, applicable to human-agent interaction including both virtual agents and robots. Specifically, we extend recent deep-learning-based, data-driven methods for speech-driven gesture generation by incorporating representation learning. Our model takes speech as input and produces gestures as output, in the form of a sequence of 3D coordinates.Our approach consists of two steps. First, we learn a lowerdimensional representation of human motion using a denoising autoencoder neural network, consisting of a motion encoder Mo-tionE and a motion decoder MotionD. The learned representation preserves the most important aspects of the human pose variation while removing less relevant variation. Second, we train a novel encoder network SpeechE to map from speech to a corresponding motion representation with reduced dimensionality. At test time, the speech encoder and the motion decoder networks are combined: SpeechE predicts motion representations based on a given speech signal and MotionD then decodes these representations to produce motion sequences.We evaluate different representation sizes in order to find the most effective dimensionality for the representation. We also evaluate the effects of using different speech features as input to the model. We find that mel-frequency cepstral coefficients (MFCCs), alone or combined with prosodic features, perform the best. The results of a subsequent user study confirm the benefits of the representation learning.

Gesticulator: A framework for semantically-aware speech-driven gesture generation

“…Audio-Driven Gesture Generation. Most prior work on datadriven gesture generation has used the audio-signal as the only speech-input modality in the model [14,15,19,28,42]. For example, Sadoughi and Busso [42] trained a probabilistic graphical model to generate a discrete set of gestures based on the speech audiosignal, using discourse functions as constraints.…”

“…Hasegawa et al [19] developed a more general model capable of generating arbitrary 3D motion using a deep recurrent neural network, applying smoothing as postprocessing step. Kucherenko et al [28] extended this work by applying representation learning to the human pose and reducing the need for smoothing. Recently, Ginosar et al [15] applied a convolutional neural network with adversarial training to generate 2D poses from spectrogram features.…”

“…As a step towards common evaluation measures for the gesture generation field, we primarily use metrics proposed by previous researchers. Specifically, we evaluated the average values of rootmean-square error (RMSE), acceleration and jerk (rate of change of acceleration), and acceleration histograms of the produced motion, in line with Kucherenko et al [28]. To obtain these statistics, the gestures were converted from joint angles to 3D joint positions.…”

On Designing Expressive Robot Behavior: The Effect of Affective Cues on Interaction

“…Hasegawa et al [34] discussed a data-driven model for metaphoric gesture motion synthesis for a stick figure based on a speech input in Japanese; however, the generated gestures were rated relatively lower than the original gestures in semantic consistency. This model was further improved through motion representation learning to ameliorate gesture motion synthesis [49] but using the same language. Yoon et al [91] introduced a data-driven endto-end robot model for generating different categories of gestures (including iconic and metaphoric gestures) based on an input text and not a direct speech, which is similar to the rule-based gesture generators explained earlier.…”

Energy flows in gesture-speech physics: Exploratory findings and pre-registration of confirmatory analysis

“…A recurrent neural network trained on person-specific gesture-speech sequences (motion and audio data from talk shows), was able to produce novel speech-synchronous gestures based on novel speech from the person the neural network was trained on (Ginosar et al, 2019). These neural networks are thus showing that there must be some person-specific invariant between speech acoustics and gesture motion, although it remains unknown what the neural network in fact picked up on in speech so as to produce gesture so well (but see Kucherenko, Hasegawa, Henter, Kaneko, & Kjellström, 2019).…”