Expressive Speech Driven Talking Avatar Synthesis with DBLSTM Using Limited Amount of Emotional Bimodal Data

Li, Xu; Wu, Zhiyong; Meng, Helen; Jia, Jia; Lou, Xiaoyan; Cai, Lianhong

doi:10.21437/interspeech.2016-364

Cited by 16 publications

(8 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The study compared the results achieved with their model with a HMM-based approach, showing improvements in terms of objective and subjective metrics. Li et al [6] proposed strategies using BLSTM models to create emotional facial movement by having access to a small emotional dataset. They proposed several approaches to leverage recordings from a neutral corpus and a small emotional corpus, aiming to improve the emotional regression result.…”

Section: Dnn-based Modelingmentioning

confidence: 99%

Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks

Sadoughi

Busso

2021

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

Articulation, emotion, and personality play strong roles in the orofacial movements. To improve the naturalness and expressiveness of virtual agents (VAs), it is important that we carefully model the complex interplay between these factors. This paper proposes a conditional generative adversarial network, called conditional sequential GAN (CSG), which learns the relationship between emotion and lexical content in a principled manner. This model uses a set of articulatory and emotional features directly extracted from the speech signal as conditioning inputs, generating realistic movements. A key feature of the approach is that it is a speech-driven framework that does not require transcripts. Our experiments show the superiority of this model over three state-of-the-art baselines in terms of objective and subjective evaluations. When the target emotion is known, we propose to create emotionally dependent models by either adapting the base model with the target emotional data (CSG-Emo-Adapted), or adding emotional conditions as the input of the model (CSG-Emo-Aware). Objective evaluations of these models show improvements for the CSG-Emo-Adapted compared with the CSG model, as the trajectory sequences are closer to the original sequences. Subjective evaluations show significantly better results for this model compared with the CSG model when the target emotion is happiness.

show abstract

Section: Dnn-based Modelingmentioning

confidence: 99%

Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks

Sadoughi

Busso

2021

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

show abstract

“…Li et al [19] used a recurrent network and compared several BLSTM architectures to adapt a model trained on a large neutral corpus with a small quantity of expressive data. The five proposed systems generate expressive visual animations from audio files.…”

Section: Related Workmentioning

confidence: 99%

“…One way to overcome this limitation is by taking advantage of the neutral data available and to link it with the emotional data. For instance, Li et al [19] used recurrent network (DBLSTM) to generate audiovisual animation from audio by simply retraining the model with emotion-specific data. Their experiments showed that using neutral corpus can improve the performance of the synthesis of expressive talking avatar animations.…”

Section: Introductionmentioning

confidence: 99%

Learning emotions latent representation with CVAE for text-driven expressive audiovisual speech synthesis

et al. 2021

View full text Add to dashboard Cite

Great improvement has been made in the field of expressive audiovisual Textto-Speech synthesis (EAVTTS) thanks to deep learning techniques. However, generating realistic speech is still an open issue and researchers in this area have been focusing lately on controlling the speech variability. In this paper, we use different neural architectures to synthesize emotional speech. We study the application of unsupervised learning techniques for emotional speech modeling as well as methods for restructuring emotions representation to make it continuous and more flexible. This manipulation of the emotional representation should allow us to generate new styles of speech by mixing emotions. We first present our expressive audiovisual corpus. We validate the emotional content of this corpus with three perceptual experiments using acoustic only, visual only and audiovisual stimuli. After that, we analyze the performance of a fully connected neural network in learning characteristics specific to different emotions for the phone duration aspect and the acoustic and visual modalities. We also study the contribution of a joint and separate training of the acoustic and visual modalities in the quality of the generated synthetic speech. In the second part of this paper, we use a conditional variational auto-encoder (CVAE) architecture to learn a latent representation of emotions. We applied this method in an unsupervised manner to generate features of expressive speech. We used a

show abstract

“…Some methods for automatic expressive 3D character animation have emerged taking advantages of the progress in the deep learning area. Xu Li et al [7] used recurrent network (DBLSTM) to generate audiovisual animation from audio by simply retraining the model with emotion-specific data. Their experiments showed that using neutral corpus can improve the performance of expressive talking avatar generation.…”

Section: Introductionmentioning

confidence: 99%

Conditional Variational Auto-Encoder for Text-Driven Expressive AudioVisual Speech Synthesis

Dahmani¹,

Colotte²,

Girard³

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

In recent years, the performance of speech synthesis systems has been improved thanks to deep learning-based models, but generating expressive audiovisual speech is still an open issue. The variational auto-encoders (VAE)s are recently proposed to learn latent representations of data. In this paper, we present a system for expressive text-to-audiovisual speech synthesis that learns a latent embedding space of emotions using a conditional generative model based on the variational auto-encoder framework. When conditioned on textual input, the VAE is able to learn an embedded representation that captures emotion characteristics from the signal, while being invariant to the phonetic content of the utterances. We applied this method in an unsupervised manner to generate duration, acoustic and visual features of speech. This conditional variational auto-encoder (CVAE) has been used to blend emotions together. This model was able to generate nuances of a given emotion or to generate new emotions that do not exist in our database. We conducted three perceptive experiments to evaluate our findings.

show abstract

Expressive Speech Driven Talking Avatar Synthesis with DBLSTM Using Limited Amount of Emotional Bimodal Data

Cited by 16 publications

References 13 publications

Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks

Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks

Learning emotions latent representation with CVAE for text-driven expressive audiovisual speech synthesis

Conditional Variational Auto-Encoder for Text-Driven Expressive AudioVisual Speech Synthesis

Contact Info

Product

Resources

About