Building a controllable expressive speech synthesis system with multiple emotion strengths

Zhu, Xiaolian; Xue, Liumeng

doi:10.1016/j.cogsys.2019.09.009

Cited by 16 publications

(12 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Naturally, there have been several attempts to obtain emotion strength information without manual works. Zhu and Xue [48] applied a K-means clustering algorithm to partition emotion strength levels for a speech corpus and developed an embedding vector that continuously represents the emotion strength using a t-distributed stochastic neighbor embedding (t-SNE) algorithm [49]. Zhu et al [50] also applied the concept of a relative attribute to learn a ranking function for each emotion category and controlled the emotion strength continuously.…”

Section: Emotion Strength Control Modellingmentioning

confidence: 99%

Sequence-to-Sequence Emotional Voice Conversion With Strength Control

Choi

Hahn

2021

IEEE Access

View full text Add to dashboard Cite

This paper proposes an improved emotional voice conversion (EVC) method with emotional strength and duration controllability. EVC methods without duration mapping generate emotional speech with identical duration to that of the neutral input speech. In reality, even the same sentences would have different speeds and rhythms depending on the emotions. To solve this, the proposed method adopts a sequence-to-sequence network with an attention module that enables the network to learn attention in the neutral input sequence should be focused on which part of the emotional output sequence. Besides, to capture the multi-attribute aspects of emotional variations, an emotion encoder is designed for transforming acoustic features into emotion embedding vectors. By aggregating the emotion embedding vectors for each emotion, a representative vector for the target emotion is obtained and weighted to reflect emotion strength. By introducing a speaker encoder, the proposed method can preserve speaker identity even after the emotion conversion. Objective and subjective evaluation results confirm that the proposed method is superior to other previous works. Especially, in emotion strength control, we achieve in getting successful results.

show abstract

Section: Emotion Strength Control Modellingmentioning

confidence: 99%

Sequence-to-Sequence Emotional Voice Conversion With Strength Control

Choi

Hahn

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

“…The first works in DNNs-based acoustic speech synthesis appeared in 2013 and used FeedForward DNNs to model the mapping between linguistic and acoustic features [12,29,30,31]. Later, other studies worked on adding expressiveness to the synthesized voice [32,20,33,34]. Regarding audiovisual speech, some works used DNNs to model emotion categories such as [35] and [16] who used FeedForward DNNs to synthesize expressive audiovisual speech.…”

Section: Related Workmentioning

confidence: 99%

“…Henter et al [39] and Zhu et al [34] succeeded in creating nuances of emotions without using emotion degree annotations, nevertheless, this work still relies on emotion labels as input.…”

Section: Related Workmentioning

confidence: 99%

Learning emotions latent representation with CVAE for text-driven expressive audiovisual speech synthesis

et al. 2021

View full text Add to dashboard Cite

Great improvement has been made in the field of expressive audiovisual Textto-Speech synthesis (EAVTTS) thanks to deep learning techniques. However, generating realistic speech is still an open issue and researchers in this area have been focusing lately on controlling the speech variability. In this paper, we use different neural architectures to synthesize emotional speech. We study the application of unsupervised learning techniques for emotional speech modeling as well as methods for restructuring emotions representation to make it continuous and more flexible. This manipulation of the emotional representation should allow us to generate new styles of speech by mixing emotions. We first present our expressive audiovisual corpus. We validate the emotional content of this corpus with three perceptual experiments using acoustic only, visual only and audiovisual stimuli. After that, we analyze the performance of a fully connected neural network in learning characteristics specific to different emotions for the phone duration aspect and the acoustic and visual modalities. We also study the contribution of a joint and separate training of the acoustic and visual modalities in the quality of the generated synthetic speech. In the second part of this paper, we use a conditional variational auto-encoder (CVAE) architecture to learn a latent representation of emotions. We applied this method in an unsupervised manner to generate features of expressive speech. We used a

show abstract

“…Cilj rada je teorijska analiza i poređenje dvije inovativne metode koje teže da modeluju kompleksnost emocija u vidu kontinualnih vektora kojima je moguće manipulisati. U prvom pristupu [5], korišćenjem kontrolnog embedding vektora kao dodatnog ulaza LSTM (eng. Long Short-Term Memory -LSTM) duboke neuronske mreže (eng.…”

Section: Uvodunclassified

Poređenje Sistema Za Sintezu Ekspresivnog Govora Sa Mogućnošću Kontrole Jačine Emocije

Vujović¹

2020

Zbornik radova FTN

View full text Add to dashboard Cite

U sintezi ekspresivnog govora važno je generisati emocionalno obojen govor koji odražava kompleksnost emocionalnih stanja. Brojni TTS sistemi emocije u sintetizovanom govoru modeluju u vidu diskretnih skupova, ali tek kada se uzmu u obzir i varijacije koje postoje unutar emotivnih stanja, generisani govor može biti nalik ljudskom. Ovaj rad obuhvata teorijsku analizu i poređenje dva inovativna sistema za sintezu ekspresivnog govora koji kompleksnost emocija modeluju u vidu kontinualnih vektora kojima je moguće manipulisati. Rezultati pokazuju da je pristup zasnovan na t-SNE embedding vektorima primjenljiv samo u slučaju specifičnih baza podataka, dok je drugi pristup, zasnovan na interpolaciji tačaka u embedding prostoru multi-speaker, multi-style modela, opštiji, ali zahtijeva dodatnu analizu.

show abstract

Building a controllable expressive speech synthesis system with multiple emotion strengths

Cited by 16 publications

References 14 publications

Sequence-to-Sequence Emotional Voice Conversion With Strength Control

Sequence-to-Sequence Emotional Voice Conversion With Strength Control

Learning emotions latent representation with CVAE for text-driven expressive audiovisual speech synthesis

Poređenje Sistema Za Sintezu Ekspresivnog Govora Sa Mogućnošću Kontrole Jačine Emocije

Contact Info

Product

Resources

About