Emotion transplantation through adaptation in HMM-based speech synthesis

Lorenzo-Trueba, Jaime; Barra-Chicote, Roberto; San-Segundo, Rubén; Ferreiros, Javier; Yamagishi, Junichi; Montero, Juan Manuel

doi:10.1016/j.csl.2015.03.008

Cited by 33 publications

(12 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their main advantage is the possibility of generating speaker-independent emotional speech models that aggregate information from different speakers and different emotions, providing highly controllable and reliable systems. Some tasks they have proved to been successful on range from emotional intensity control of 2 emotional classes (joyful and sad) (Nose and Kobayashi, 2012) to emotion transplantation of 4 emotional classes (angry, happy, sad and surprised) (Lorenzo-Trueba et al, 2015).…”

Section: Overview Of Emotional Speech Synthesismentioning

confidence: 99%

Investigating different representations for modeling and controlling multiple emotions in DNN-based speech synthesis

Lorenzo-Trueba

Henter

Takaki

et al. 2018

Speech Communication

Self Cite

View full text Add to dashboard Cite

In this paper, we investigate the simultaneous modeling of multiple emotions in DNN-based expressive speech synthesis, and how to represent the emotional labels, such as emotional class and strength, for this task. Our goal is to answer two questions: First, what is the best way to annotate speech data with multiple emotions-should we use the labels that the speaker intended to express, or labels based on listener perception of the resulting speech signals? Second, how should the emotional information be represented as labels for supervised DNN training, e.g., should emotional class and emotional strength be factorized into separate inputs or not? We evaluate on a large-scale corpus of emotional speech from a professional voice actress, additionally annotated with perceived emotional labels from crowdsourced listeners. By comparing DNN-based speech synthesizers that utilize different emotional representations, we assess the impact of these representations and design decisions on human emotion recognition rates, perceived emotional strength, and subjective speech quality. Simultaneously, we also study which representations are most appropriate for controlling the emotional strength of synthetic speech.

show abstract

Section: Overview Of Emotional Speech Synthesismentioning

confidence: 99%

Investigating different representations for modeling and controlling multiple emotions in DNN-based speech synthesis

Lorenzo-Trueba

Henter

Takaki

et al. 2018

Speech Communication

Self Cite

View full text Add to dashboard Cite

show abstract

“…speech rate, speech intensity, F0, and F0 range), the HTS-et synthesizer was chosen for the parametric modelling of emotions. In addition to a parametric synthesis of emotional speech, this statistical parametric synthesis method is also suitable for a corpus-based approach; for instance, corpora of emotional speech are used to train speech models (Yamagishi et al 2005, Lorenzo-Trueba et al 2015. We used two synthetic voices trained on neutral speech, a male voice, Tõnu, and a female voice, Eva, as the basic voices of the Estonian HTS synthesis.…”

Section: Estonian Speech Synthesizersmentioning

confidence: 99%

Expression of basic emotions in Estonian parametric text-to-speech synthesis

Tamuri

Mihkla

2015

ESUKA-JEFUL

View full text Add to dashboard Cite

The goal of this study was to conduct modelling experiments, the purpose of which was the expression of three basic emotions (joy, sadness and anger) in Estonian parametric text-to-speech synthesis on the basis of both a male and a female voice. For each emotion, three different test models were constructed and presented for evaluation to subjects in perception tests. The test models were based on the basic emotions’ characteristic parameter values that had been determined on the basis of human speech. In synthetic speech, the test subjects most accurately recognized the emotion of sadness, and least accurately the emotion of joy. The results of the test showed that, in the case of the synthesized male voice, the model with enhanced parameter values performed best for all three emotions, whereas in the case of the synthetic female voice, different emotions called for different models: the model with decreased values was the most suitable one for the expression of joy, and the model with enhanced values was the most suitable for the expression of sadness and anger. Logistic regression was applied to the results of the perception tests in order to determine the significance and contribution of each acoustic parameter in the emotion models, and the possible need to adjust the values of the parameters.Kokkuvõte. Kairi Tamuri ja Meelis Mihkla: Põhiemotsioonide väljendusvõimalused eestikeelsel parameetrilisel kõnesünteesil. Uurimistöö eesmärk oli läbi viia modelleerimiseksperimente kolme põhiemotsiooni (rõõmu, kurbuse ja viha) väljendamiseks eestikeelsel parameetrilisel kõnesünteesil nii mees- kui ka naissünteeshääle baasil. Selleks koostati iga emotsiooni kohta kolm erinevat katsemudelit, mida lasti katseisikutel tajutestidel hinnata. Katsemudelite aluseks oli inimkõne põhjal määratud põhiemotsioonidele omased parameetrite väärtused. Emotsioonidest tunti sünteeskõnes kõige paremini ära kurbuse-emotsioon ning kõige halvemini rõõmu-emotsioon. Testitulemused näitasid, et kui meessünteeshääle puhul töötas kõigi kolme emotsiooni puhul kõige paremini võimendatud väärtuste mudel, siis naissünteeshääle puhul vajasid erinevad emotsioonid erinevaid mudeleid: rõõmu väljendamiseks sobis kõige paremini vähendatud väärtuste mudel, kurbuse ja viha väljendamiseks võimendatud väärtuste mudel. Tajutestide tulemusi analüüsiti logistilisel regressioonil, et teha kindlaks üksikute akustiliste parameetrite olulisus ja osakaal emotsiooni mudelites ning parameetrite väärtuste korrigeerimisvajadused.Märksõnad: eesti keel, emotsioonid, kõnesüntees, akustiline mudel, kõnetempo, intensiivsus, põhitoon

show abstract

“…Only a few adaptation strategies have been tested extensively by emotion-synthesis researchers. Most of these strategies were based on a hidden Markov model using a constrained structural maximum a posteriori linear regression (CSMAPLR) adaptation [51] or training a speaker-dependent model by adapting from speakerindependent average models [52]. In addition, the method in [52] used a segment of duration as short as 5 min of the emotional data from the target speaker and could produce an appreciable perception of the synthesized emotion, however compromising on the speech quality.…”

Section: Introductionmentioning

confidence: 99%

Emotional Voice Conversion Using a Hybrid Framework With Speaker-Adaptive DNN and Particle-Swarm-Optimized Neural Network

et al. 2020

View full text Add to dashboard Cite

We propose a hybrid network-based learning framework for speaker-adaptive vocal emotion conversion, tested on three different datasets (languages), namely, EmoDB (German), IITKGP (Telugu), and SAVEE (English). The optimized learning model introduced is unique because of its ability to synthesize emotional speech with an acceptable perceptive quality while preserving speaker characteristics. The multilingual model is extremely beneficial in scenarios wherein emotional training data from a specific target speaker are sparsely available. The proposed model uses speaker-normalized mel-generalized cepstral coefficients for spectral training with data adaptation using the seed data from the target speaker. The fundamental frequency (F0) is transformed using a wavelet synchrosqueezed transform prior to mapping to obtain a sharpened time-frequency representation. Moreover, a feedforward artificial neural network, together with particle swarm optimization, was used for F0 training. Additionally, static-intensity modification was also performed for each test utterance. Using the framework, we were able to capture the spectral and pitch contour variabilities of emotional expression better than with other state-of-the-art methods used in this study. Considering the overall performance scores across datasets, an average melcepstral distortion (MCD) of 4.98 and root mean square error (RMSE-F0) of 10.67 were obtained in objective evaluations, and an average comparative mean opinion score (CMOS) of 3.57 and speaker similarity score of 3.70 were obtained for the proposed framework. Particularly, the best MCD of 4.09 (EmoDB-happiness) and RMSE-F0 of 9.00 (EmoDB-anger) were obtained, along with the maximum CMOS of 3.7 and speaker similarity of 4.6, thereby highlighting the effectiveness of the hybrid network model.

show abstract

Emotion transplantation through adaptation in HMM-based speech synthesis

Cited by 33 publications

References 20 publications

Investigating different representations for modeling and controlling multiple emotions in DNN-based speech synthesis

Investigating different representations for modeling and controlling multiple emotions in DNN-based speech synthesis

Expression of basic emotions in Estonian parametric text-to-speech synthesis

Emotional Voice Conversion Using a Hybrid Framework With Speaker-Adaptive DNN and Particle-Swarm-Optimized Neural Network

Contact Info

Product

Resources

About