End-to-End Emotional Speech Synthesis Using Style Tokens and Semi-Supervised Training

Wu, Pengfei; Ling, Zhen-Hua; Liu, Lijuan; Jiang, Yuan; Hongchuan, Wu; Dai, Li-Rong

doi:10.1109/apsipaasc47483.2019.9023186

Cited by 53 publications

(37 citation statements)

References 12 publications

(16 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For synthetic speech quality, we conducted objective experiments. We used the root mean square error of F0 (F0RMSE) and melcepstrum distortions (MCD) as objective metrics, as in [22]. The FASTDTW [23] algorithm was adopted to align the predicted acoustic feature sequences with the natural ones.…”

Section: Accuracy Of Acoustic Feature Predictionmentioning

confidence: 99%

Improving Naturalness and Controllability of Sequence-to-Sequence Speech Synthesis by Learning Local Prosody Representations

Gong

Wang

Ling

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

State-of-the-art neural text-to-speech (TTS) networks are trained with a large amount of speech data, which significantly improves the quality of synthetic speech compared with traditional approaches. However, the prosody and controllability of the generated speech is still insufficient, especially in tonal languages. Moreover, the generated prosody is solely defined by the input text, which does not allow for different styles for the same sentence or words. In this study, we extended Tacotron2 with a pitch prediction task to capture discrete pitch-related representations. Specifically, the learned pitch-related suprasegmental information is fed simultaneously with traditional character features into the decoder to generate final Mel spectrogram. Experiments show that the proposed method can improve the quality of the generated speech (mean opinion score of 4.37 vs. 4.22). Moreover, we demonstrated that we can easily achieve wordlevel pitch control during generation by changing local pitch-related representations before passing them to the decoder network.

show abstract

Section: Accuracy Of Acoustic Feature Predictionmentioning

confidence: 99%

Improving Naturalness and Controllability of Sequence-to-Sequence Speech Synthesis by Learning Local Prosody Representations

Gong

Wang

Ling

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…There have been studies to leverage emotion speech modeling for expressive TTS [33], [68], [86]- [88]. Eyben et al [68] incorporate unsupervised expression cluster information into an HMM-based TTS system.…”

Section: Deep Features For Perceptual Lossmentioning

confidence: 99%

Expressive TTS Training With Frame and Style Reconstruction Loss

Liu

Şişman

Gao

et al. 2021

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

We propose a novel training strategy for Tacotronbased text-to-speech (TTS) system that improves the speech styling at utterance level. One of the key challenges in prosody modeling is the lack of reference that makes explicit modeling difficult. The proposed technique doesn't require prosody annotations from training data. It doesn't attempt to model prosody explicitly either, but rather encodes the association between input text and its prosody styles using a Tacotron-based TTS framework. This study marks a departure from the style token paradigm where prosody is explicitly modeled by a bank of prosody embeddings. It adopts a combination of two objective functions: 1) frame level reconstruction loss, that is calculated between the synthesized and target spectral features; 2) utterance level style reconstruction loss, that is calculated between the deep style features of synthesized and target speech. The style reconstruction loss is formulated as a perceptual loss to ensure that utterance level speech style is taken into consideration during training. Experiments show that the proposed training strategy achieves remarkable performance and outperforms the state-ofthe-art baseline in both naturalness and expressiveness. To our best knowledge, this is the first study to incorporate utterance level perceptual quality as a loss function into Tacotron training for improved expressiveness.

show abstract

“…At last, a weighted sum of GST is used as the style embedding. Numerous GST-Tacotron variants [11,20] have been proposed to improve model performance. However, most of them are no longer unsupervised methods.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Learning for Multi-Style Speech Synthesis with Limited Data

Liang

Cao

Chen

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Existing multi-style speech synthesis methods require either style labels or large amounts of unlabeled training data, making data acquisition difficult. In this paper, we present an unsupervised multi-style speech synthesis method that can be trained with limited data. We leverage instance discriminator to guide a style encoder to learn meaningful style representations from a multi-style dataset. Furthermore, we employ information bottleneck to filter out style-irrelevant information in the representations, which can improve speech quality and style similarity. Our method is able to produce desirable speech using a fairly small dataset, where the baseline GST-Tacotron fails. ABX tests show that our model significantly outperforms GST-Tacotron in both emotional speech synthesis task and multi-speaker speech synthesis task. In addition, we demonstrate that our method is able to learn meaningful style features with only 50 training samples per style.

show abstract

End-to-End Emotional Speech Synthesis Using Style Tokens and Semi-Supervised Training

Cited by 53 publications

References 12 publications

Improving Naturalness and Controllability of Sequence-to-Sequence Speech Synthesis by Learning Local Prosody Representations

Improving Naturalness and Controllability of Sequence-to-Sequence Speech Synthesis by Learning Local Prosody Representations

Expressive TTS Training With Frame and Style Reconstruction Loss

Unsupervised Learning for Multi-Style Speech Synthesis with Limited Data

Contact Info

Product

Resources

About