Towards Multimodal Human-Like Characteristics and Expressive Visual Prosody in Virtual Agents

Fares, Mireille

doi:10.1145/3382507.3421155

Cited by 8 publications

(7 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…and Convolutional Neural Networks (CNNs) combined with GANs [16] were used to pblackict speech-driven head motion, facial, hand and body gestures. Other works driven by both speech and text semantics were proposed for upper-facial [11], [9], [10] and hand [19] gesture generation, however they cannot be used for style modelling and control. Synthesizing expressive gestures while controlling their style is recently receiving more attention [3], [17], [8], [14].…”

Section: Related Work and Our Contributionsmentioning

confidence: 99%

See 1 more Smart Citation

Zero-Shot Style Transfer for Multimodal Data-Driven Gesture Synthesis

Fares

Obin

Pélachaud

2023

2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG)

Self Cite

View full text Add to dashboard Cite

We propose a multimodal speech driven approach to generate 2D upper-body gestures for virtual agents, in the communicative style of different speakers, seen or unseen by our model during training. Upper-body gestures of a source speaker are generated based on the content of his/her multimodal data -speech acoustics and text semantics. The synthesized source speaker's gestures are conditioned on the multimodal style representation of the target speaker. Our approach is zero-shot, and can generalize the style transfer to new unseen speakers, without any additional training. An objective evaluation is conducted to validate our approach.

show abstract

Section: Related Work and Our Contributionsmentioning

confidence: 99%

“…propose a probabilistic approach based on normalizing flows for synthesizing facial gestures in dyadic settings. Facial (Fares et al [2021b], Fares [2020]) and hand ) gestures driven by both acoustic and semantic information are the closest approaches to our gesture generation task, however they cannot be used for the style transfer task.…”

Section: Related Workmentioning

confidence: 99%

Zero-Shot Style Transfer for Multimodal Data-Driven Gesture Synthesis

Fares

Obin

Pélachaud

2023

2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Furthermore, transformer networks and attention mechanisms were recently used for upper-facial gesture synthesis based on multimodal data—text and speech (Fares et al, 2021b ). Facial (Fares, 2020 ; Fares et al, 2021b ) and hand (Kucherenko et al, 2020 ) gestures driven by both acoustic and semantic information are the closest approaches to our gesture generation task; however, they cannot be used for the style transfer task.…”

Section: Related Workmentioning

confidence: 99%

“…LSTM networks driven by speech were recently used to predict sequences of gestures (Hasegawa et al, 2018) and body motions (Shlizerman et al, 2018;Ahuja et al, 2019). LSTMs were additionally employed for synthesizing sequences of facial gestures driven by text and speech, namely, the fundamental frequency (F0) (Fares, 2020;Fares et al, 2021a). Generative adversarial networks (GANs) were proposed to generate realistic head motion (Sadoughi and Busso, 2018) and body motions (Ferstl et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

“…Jonell et al (2020) proposed a probabilistic approach based on normalizing flows for synthesizing facial gestures in dyadic settings. Facial (Fares, 2020;Fares et al, 2021b) and hand (Kucherenko et al, 2020) gestures driven by both acoustic and semantic information are the closest approaches to our gesture generation task; however, they cannot be used for the style transfer task.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Zero-shot style transfer for gesture animation driven by text and speech using adversarial disentanglement of multimodal style encoding

Fares¹,

Pélachaud²,

Obin³

2023

Front. Artif. Intell.

Self Cite

View full text Add to dashboard Cite

Modeling virtual agents with behavior style is one factor for personalizing human-agent interaction. We propose an efficient yet effective machine learning approach to synthesize gestures driven by prosodic features and text in the style of different speakers including those unseen during training. Our model performs zero-shot multimodal style transfer driven by multimodal data from the PATS database containing videos of various speakers. We view style as being pervasive; while speaking, it colors the communicative behaviors expressivity while speech content is carried by multimodal signals and text. This disentanglement scheme of content and style allows us to directly infer the style embedding even of a speaker whose data are not part of the training phase, without requiring any further training or fine-tuning. The first goal of our model is to generate the gestures of a source speaker based on the content of two input modalities–Mel spectrogram and text semantics. The second goal is to condition the source speaker's predicted gestures on the multimodal behavior style embedding of a target speaker. The third goal is to allow zero-shot style transfer of speakers unseen during training without re-training the model. Our system consists of two main components: (1) a speaker style encoder network that learns to generate a fixed-dimensional speaker embedding style from a target speaker multimodal data (mel-spectrogram, pose, and text) and (2) a sequence-to-sequence synthesis network that synthesizes gestures based on the content of the input modalities—text and mel-spectrogram—of a source speaker and conditioned on the speaker style embedding. We evaluate that our model is able to synthesize gestures of a source speaker given the two input modalities and transfer the knowledge of target speaker style variability learned by the speaker style encoder to the gesture generation task in a zero-shot setup, indicating that the model has learned a high-quality speaker representation. We conduct objective and subjective evaluations to validate our approach and compare it with baselines.

show abstract