Seen and Unseen Emotional Style Transfer for Voice Conversion with A New Emotional Speech Dataset

Zhou, Kun; Şişman, Berrak; Li, Rui; Li, Haizhou

doi:10.1109/icassp39728.2021.9413391

Cited by 109 publications

(55 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use a multi-speaker emotional speech dataset, ESD [14], to conduct all the experiments. ESD consists of multi-lingual and multi-speaker parallel emotional speech data with five emotions (neutral, happy, sad, angry and surprise), and has been used in emotional voice conversion [49,53] and emotional text-to-speech [54].…”

Section: Methodsmentioning

confidence: 99%

“…Many studies [46,47] have shown that the deep features learned by DNN are more effective and thus more suitable for SER. Meanwhile, recent speech synthesis studies [48,49] also propose to leverage those deep emotional features to characterize different emotional styles over a continuum [50]. These successful attempts have served as the source of motivation for this paper.…”

Section: Speaker-dependent Emotional Stylementioning

confidence: 99%

“…As emotional style presents both speaker-dependent and speaker-independent characteristics at the same time, it is insufficient for us to use a discrete representation to represent different emotional styles, such as one-hot emotion label [49]. Therefore, we propose to use deep emotional features that learnt from a large emotional speech corpus to describe different emotional styles.…”

Section: Stage I: Emotional Style Descriptor Trainingmentioning

confidence: 99%

See 2 more Smart Citations

Expressive Voice Conversion: A Joint Framework for Speaker Identity and Emotional Style Transfer

Du¹,

Şişman²,

Zhou³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Traditional voice conversion (VC) has been focused on speaker identity conversion for speech with a neutral expression. We note that emotional expression plays an essential role in daily communication, and the emotional style of speech can be speaker-dependent. In this paper, we study the technique to jointly convert the speaker identity and speakerdependent emotional style, that is called expressive voice conversion. We propose a StarGAN-based framework to learn a many-to-many mapping across different speakers, that takes into account speaker-dependent emotional style without the need for parallel data. To achieve this, we condition the generator on emotional style encoding derived from a pre-trained speech emotion recognition (SER) model. The experiments validate the effectiveness of our proposed framework in both objective and subjective evaluations. To our best knowledge, this is the first study on expressive voice conversion.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Speaker-dependent Emotional Stylementioning

confidence: 99%

Section: Stage I: Emotional Style Descriptor Trainingmentioning

confidence: 99%

See 1 more Smart Citation

Expressive Voice Conversion: A Joint Framework for Speaker Identity and Emotional Style Transfer

Du¹,

Şişman²,

Zhou³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We first give a comprehensive overview of recent studies on emotional voice conversion in Section 2. We discuss the 15 existing databases [68] in Section 3. We then formulate the design of a novel ESD database for speaker-independent emotional voice conversion, that is also suitable for other speech synthesis tasks, such as mono-lingual or cross-lingual speaker voice conversion and emotional text-to-speech.…”

Section: Speaker a (Happy)mentioning

confidence: 99%

Emotional Voice Conversion: Theory, Databases and ESD

Zhou¹,

Şişman²,

Li³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we first provide a review of the state-of-the-art emotional voice conversion research, and the existing emotional speech databases. We then motivate the development of a novel emotional speech database (ESD) that addresses the increasing research need. With this paper, the ESD database 1 is now made available to the research community. The ESD database consists of 350 parallel utterances spoken by 10 native English and 10 native Chinese speakers and covers 5 emotion categories (neutral, happy, angry, sad and surprise). More than 29 hours of speech data were recorded in a controlled acoustic environment. The database is suitable for multi-speaker and cross-lingual emotional voice conversion studies. As case studies, we implement several state-of-the-art emotional voice conversion systems on the ESD database. This paper provides a reference study on ESD in conjunction with its release.

show abstract

“…The (Kun et al, 2021). The RAVDESS corpus is an audio-visual database of emotional speech and songs.…”

Section: Formation Of An Emotional Corpus Of Speech Signalsmentioning

confidence: 99%

Aviation Profiling Method Based on Deep Learning Technology for Emotion Recognition by Speech Signal

Кошеков

Savostin

Anayatova

et al. 2021

Transport and Telecommunication Journal

View full text Add to dashboard Cite

This paper proposes a method of automatic speaker-independent recognition of human psycho-emotional states by analyzing the speech signal based on Deep Learning technology to solve the problems of aviation profiling. For this purpose, an algorithm to classify seven human psycho-emotional states, including anger, joy, fear, surprise, disgust, sadness, and neutral state was developed. The algorithm is based on the use of Mel-frequency cepstral coefficients and Mel spectrograms as informative features of speech signals audio recordings. These informative features are used to train two deep convolutional neural networks on the generated dataset. The developed classifier testing on a delayed verification dataset showed that the metric for the multiclass fraction of correct answers’ accuracy is 0.93. The solution proposed in the paper can be in demand in human-machine interfaces creation, medicine, marketing, and in the field of air transportation.

show abstract

Seen and Unseen Emotional Style Transfer for Voice Conversion with A New Emotional Speech Dataset

Cited by 109 publications

References 26 publications

Expressive Voice Conversion: A Joint Framework for Speaker Identity and Emotional Style Transfer

Expressive Voice Conversion: A Joint Framework for Speaker Identity and Emotional Style Transfer

Emotional Voice Conversion: Theory, Databases and ESD

Aviation Profiling Method Based on Deep Learning Technology for Emotion Recognition by Speech Signal

Contact Info

Product

Resources

About