Creating a Gesture-Speech Dataset for Speech-Based Automatic Gesture Generation

Takeuchi, Kenta; Kubota, Shin; Suzuki, Keisuke; Hasegawa, Dai; Sakuta, Hiroshi

doi:10.1007/978-3-319-58750-9_28

Cited by 29 publications

(8 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The most important goal in gesture generation is to produce motion patterns that are convincing to human observers. Since improvements in objective measures do not always translate into superior subjective quality for human observers, we validated our conclusions above through a number of user studies: two on the Japanese dataset from Takeuchi, Kubota, et al (2017) and one on the English dataset of Ferstl and McDonnell (2018). All of them used the same questionnaire as in the baseline paper Hasegawa et al (2018), shown in Table 2.…”

Section: Subjective Evaluation and Discussionmentioning

confidence: 77%

Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation

Kucherenko,

Hasegawa,

Kaneko

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

This paper presents a novel framework for speech-driven gesture production, applicable to virtual agents to enhance human-computer interaction. Specifically, we extend recent deep-learning-based, data-driven methods for speech-driven gesture generation by incorporating representation learning. Our model takes speech as input and produces gestures as output, in the form of a sequence of 3D coordinates.We provide an analysis of different representations for the input (speech) and the output (motion) of the network by both objective and subjective evaluations. We also analyse the importance of smoothing of the produced motion.Our results indicated that the proposed method improved on our baseline in terms of objective measures. For example, it better captured the motion dynamics and better matched the motion-speed distribution. Moreover, we performed user studies on two different datasets. The studies confirmed that our proposed method is perceived as more natural than the baseline, although the difference in the studies was eliminated by appropriate post-processing: hip-centering and smoothing. We conclude that it is important to take both feature representation, model architecture and post-processing into account when designing an automatic gesture-production method.

show abstract

Section: Subjective Evaluation and Discussionmentioning

confidence: 77%

Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation

Kucherenko,

Hasegawa,

Kaneko

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…There are two main methods for obtaining motion data for gesture synthesis: optical motion capture [MYL*16, TKS*17, LDM*19, JSCS19, FM18] or pose estimation from monocular video [YKJ*19, ALIM20, JKEB19, KNN*22, HXM*21].…”

Section: Data‐driven Approachesmentioning

confidence: 99%

A Comprehensive Review of Data‐Driven Co‐Speech Gesture Generation

Nyatsanga

Kucherenko²,

Ahuja³

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

Gestures that accompany speech are an essential part of natural and efficient embodied human communication. The automatic generation of such co‐speech gestures is a long‐standing problem in computer animation and is considered an enabling technology for creating believable characters in film, games, and virtual social spaces, as well as for interaction with social robots. The problem is made challenging by the idiosyncratic and non‐periodic nature of human co‐speech gesture motion, and by the great diversity of communicative functions that gestures encompass. The field of gesture generation has seen surging interest in the last few years, owing to the emergence of more and larger datasets of human gesture motion, combined with strides in deep‐learning‐based generative models that benefit from the growing availability of data. This review article summarizes co‐speech gesture generation research, with a particular focus on deep generative models. First, we articulate the theory describing human gesticulation and how it complements speech. Next, we briefly discuss rule‐based and classical statistical gesture synthesis, before delving into deep learning approaches. We employ the choice of input modalities as an organizing principle, examining systems that generate gestures from audio, text and non‐linguistic input. Concurrent with the exposition of deep learning approaches, we chronicle the evolution of the related training data sets in terms of size, diversity, motion quality, and collection method (e.g., optical motion capture or pose estimation from video). Finally, we identify key research challenges in gesture generation, including data availability and quality; producing human‐like motion; grounding the gesture in the co‐occurring speech in interaction with other speakers, and in the environment; performing gesture evaluation; and integration of gesture synthesis into applications. We highlight recent approaches to tackling the various key challenges, as well as the limitations of these approaches, and point toward areas of future development.

show abstract

“…For our experiments, we used a gesture-speech dataset collected by Takeuchi et al [36]. Motion data were recorded in a motion capture studio from two Japanese individuals having a conversation in the form of an interview.…”

Section: Gesture-speech Datasetmentioning

confidence: 99%

Analyzing Input and Output Representations for Speech-Driven Gesture Generation

Kucherenko

Hasegawa

Henter

et al. 2019

Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents

Self Cite

131

View full text Add to dashboard Cite

This paper presents a novel framework for automatic speech-driven gesture generation, applicable to human-agent interaction including both virtual agents and robots. Specifically, we extend recent deep-learning-based, data-driven methods for speech-driven gesture generation by incorporating representation learning. Our model takes speech as input and produces gestures as output, in the form of a sequence of 3D coordinates.Our approach consists of two steps. First, we learn a lowerdimensional representation of human motion using a denoising autoencoder neural network, consisting of a motion encoder Mo-tionE and a motion decoder MotionD. The learned representation preserves the most important aspects of the human pose variation while removing less relevant variation. Second, we train a novel encoder network SpeechE to map from speech to a corresponding motion representation with reduced dimensionality. At test time, the speech encoder and the motion decoder networks are combined: SpeechE predicts motion representations based on a given speech signal and MotionD then decodes these representations to produce motion sequences.We evaluate different representation sizes in order to find the most effective dimensionality for the representation. We also evaluate the effects of using different speech features as input to the model. We find that mel-frequency cepstral coefficients (MFCCs), alone or combined with prosodic features, perform the best. The results of a subsequent user study confirm the benefits of the representation learning.

show abstract

Creating a Gesture-Speech Dataset for Speech-Based Automatic Gesture Generation

Cited by 29 publications

References 4 publications

Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation

Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation

A Comprehensive Review of Data‐Driven Co‐Speech Gesture Generation

Analyzing Input and Output Representations for Speech-Driven Gesture Generation

Contact Info

Product

Resources

About