Emotional voice conversion using deep neural networks with MCC and F0 features

Luo, Zhaojie; Takiguchi, Tetsuya; Ariki, Yasuo

doi:10.1109/icis.2016.7550889

Cited by 41 publications

(38 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…RMSE is an established metric used [33], [34], [37], [49], [50], [52], [57] to evaluate the proximity of predicted values by a mapping algorithm to those of the target, given as…”

Section: ) Root Mean Square Errormentioning

confidence: 99%

“…Spectral modelling uses mostly non-linear MGCEPs. Recent advances include unsupervised training with conditional restricted Boltzmann machine (CRBM) [36], pre-training using deep belief networks (DBNs) [34], modelling the spectrum and prosody simultaneously via bidirectional long short-term memory (LSTM) [37], end-to-end emotional-speech synthesis using Tacotron [38], among others.…”

Section: Introductionmentioning

confidence: 99%

“…The F0 features for training are generally extracted using the STRAIGHT vocoder [43]. However, these features being lower dimensional, cannot be adequately converted using a DNN [32] or DBN [34]. Therefore, F0 features are frequently converted using linear-transformation-based methods, such as the logarithm Gaussian (LG) method [44].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Emotional Voice Conversion Using a Hybrid Framework With Speaker-Adaptive DNN and Particle-Swarm-Optimized Neural Network

et al. 2020

View full text Add to dashboard Cite

We propose a hybrid network-based learning framework for speaker-adaptive vocal emotion conversion, tested on three different datasets (languages), namely, EmoDB (German), IITKGP (Telugu), and SAVEE (English). The optimized learning model introduced is unique because of its ability to synthesize emotional speech with an acceptable perceptive quality while preserving speaker characteristics. The multilingual model is extremely beneficial in scenarios wherein emotional training data from a specific target speaker are sparsely available. The proposed model uses speaker-normalized mel-generalized cepstral coefficients for spectral training with data adaptation using the seed data from the target speaker. The fundamental frequency (F0) is transformed using a wavelet synchrosqueezed transform prior to mapping to obtain a sharpened time-frequency representation. Moreover, a feedforward artificial neural network, together with particle swarm optimization, was used for F0 training. Additionally, static-intensity modification was also performed for each test utterance. Using the framework, we were able to capture the spectral and pitch contour variabilities of emotional expression better than with other state-of-the-art methods used in this study. Considering the overall performance scores across datasets, an average melcepstral distortion (MCD) of 4.98 and root mean square error (RMSE-F0) of 10.67 were obtained in objective evaluations, and an average comparative mean opinion score (CMOS) of 3.57 and speaker similarity score of 3.70 were obtained for the proposed framework. Particularly, the best MCD of 4.09 (EmoDB-happiness) and RMSE-F0 of 9.00 (EmoDB-anger) were obtained, along with the maximum CMOS of 3.7 and speaker similarity of 4.6, thereby highlighting the effectiveness of the hybrid network model.

show abstract

“…RMSE is an established metric used [33], [34], [37], [49], [50], [52], [57] to evaluate the proximity of predicted values by a mapping algorithm to those of the target, given as…”

Section: ) Root Mean Square Errormentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Emotional Voice Conversion Using a Hybrid Framework With Speaker-Adaptive DNN and Particle-Swarm-Optimized Neural Network

et al. 2020

View full text Add to dashboard Cite

show abstract

“…그리하여 심층 신경망이 SPSS 방식에서 언어 특징과 음향 특징 사이의 관계를 학습하기 위한 음향 모델링의 대안으로 부상하고 있다 (Weijters & Thole, 1993;Zen et al, 2013). 또한, 음성합성에 필요한 지속시간을 예 측하기 위한 모델링 방법으로 심층 신경망을 이용할 수 있다 (Riedi, 1995 (Kubichek, 1993;Luo et al, 2016).…”

Section: 서론unclassified

Performance comparison of various deep neural network architectures using Merlin toolkit for a Korean TTS system*

Hong¹,

Kwon²

2019

Phonetics Speech Sci.

View full text Add to dashboard Cite

In this paper, we construct a Korean text-to-speech system using the Merlin toolkit which is an open source system for speech synthesis. In the text-to-speech system, the HMM-based statistical parametric speech synthesis method is widely used, but it is known that the quality of synthesized speech is degraded due to limitations of the acoustic modeling scheme that includes context factors. In this paper, we propose an acoustic modeling architecture that uses deep neural network technique, which shows excellent performance in various fields. Fully connected deep feedforward neural network (DNN), recurrent neural network (RNN), gated recurrent unit (GRU), long short-term memory (LSTM), bidirectional LSTM (BLSTM) are included in the architecture. Experimental results have shown that the performance is improved by including sequence modeling in the architecture, and the architecture with LSTM or BLSTM shows the best performance. It has been also found that inclusion of delta and delta-delta components in the acoustic feature parameters is advantageous for performance improvement.

show abstract

“…The LG-based method is insufficient to convert the prosody effectively owing to constraints of their linear models and low-dimensional F0 features. In our earlier work [16], we proposed a new NN-based method that can train the segmental F0 features for emotional prosody conversion. Although we conducted segmental processing to increase the dimensions of F0 features that can be trained by the NNs well, the segmental F0 features cannot model F0 in different temporal scales.…”

mentioning

confidence: 99%

Emotional voice conversion using neural networks with arbitrary scales F0 based on wavelet transform

Luo

Chen

Takiguchi

et al. 2017

J AUDIO SPEECH MUSIC PROC.

Self Cite

View full text Add to dashboard Cite

An artificial neural network is an important model for training features of voice conversion (VC) tasks. Typically, neural networks (NNs) are very effective in processing nonlinear features, such as Mel Cepstral Coefficients (MCC), which represent the spectrum features. However, a simple representation of fundamental frequency (F0) is not enough for NNs to deal with emotional voice VC. This is because the time sequence of F0 for an emotional voice changes drastically. Therefore, in our previous method, we used the continuous wavelet transform (CWT) to decompose F0 into 30 discrete scales, each separated by one third of an octave, which can be trained by NNs for prosody modeling in emotional VC. In this study, we propose the arbitrary scales CWT (AS-CWT) method to systematically capture F0 features of different temporal scales, which can represent different prosodic levels ranging from micro-prosody to sentence levels. Meanwhile, the proposed method uses deep belief networks (DBNs) to pre-train the NNs that then convert spectral features. By utilizing these approaches, the proposed method can change the spectrum and the F0 for an emotional voice simultaneously as well as outperform other state-of-the-art methods in terms of emotional VC.

show abstract

Emotional voice conversion using deep neural networks with MCC and F0 features

Cited by 41 publications

References 13 publications

Emotional Voice Conversion Using a Hybrid Framework With Speaker-Adaptive DNN and Particle-Swarm-Optimized Neural Network

Emotional Voice Conversion Using a Hybrid Framework With Speaker-Adaptive DNN and Particle-Swarm-Optimized Neural Network

Performance comparison of various deep neural network architectures using Merlin toolkit for a Korean TTS system*

Emotional voice conversion using neural networks with arbitrary scales F0 based on wavelet transform

Contact Info

Product

Resources

About