SRG³: Speech-driven Robot Gesture Generation with GAN

Abstract

Fig. 1: Our robot gesture generation pipeline. The speech audio is used as an input to the 3D pose synthesizer with a random noise. Then, the natural and human-like 3D gesture sequences (in the joint position(x, y, z) space) are generated. Moreover, one same speech audio with multiple different random noises can align multiple natural gesture expressions in the same manner as humans have similar but different gestures while expressing the same speech in different contexts and situations. The gesture retargeting part maps the obtained gestures in the joint position space of the Pepper robot (i.e., in the joint angle(pitch, roll, yaw) space). Finally, the mapped positions are applied on the real Pepper robot in the human-robot interaction.

A Social Robot Assisting in Cognitive Stimulation Therapy

“…Then, a training step of objective numerical errors is performed. Also in 2020, Yu et al [206] proposed a speech-driven generation method that maps a representation of speech audio to a set of appropriate gestures using a Generative Adversarial Network architecture. It has the advantage of being able to generate multiple gesture patterns for a single speech input using random noises.…”

“…Other approaches include the use of prosodic features, as shown by the work of Chiu et al [190], or directly encoding the audio signal. This last method appears in the approaches presented by Yu et al [206] or Ahuja et al [200]. Some works combine both audio features and text to improve the obtained results.…”

“…Reference Algorithm used Gesture design Multimodality [206] Encoder: 1D CNN + GRU, Decoder: GRU + MLP. Discriminator: 1D CNN + GRU + MLP Sequence of 3D poses Audio signal Table 5.1: Comparison among the works presented in the state of the art review.…”

“…Among these, a common architecture for this task is to treat the generation of expressions as a sequence-to-sequence problem, where given an encoded input sequence, the system decodes an appropriate output sequence. Authors like Ahuja et al [204], Yoon et al [195], or Yu et al [206] followed this approach. Recurrent Neural Networks tend to be used to model the encoders and decoders in this systems, due to their ability to display temporal dynamic behaviour.…”

“…Regarding the training of the model, adversarial strategies are fairly common in the works reviewed in this section. Examples of these strategies can be observed in the works of Ferstl et al [205], Ginosar et al [199], Yoon et al [113], or Yu et al [206].…”