Towards Multi-Scale Style Control for Expressive Speech Synthesis

Li, Xiang; Song, Changhe; Li, Jingbei; Wu, Zhiyong; Jia, Jia; Meng, Helen

doi:10.48550/arxiv.2104.03521

Cited by 4 publications

(8 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this subsection, we review techniques on disentangling [218,117,275], controlling [353,180,240,13,267,343,192], and transferring [150,131,393,6] variation information, as shown in Table 15. [303,377,139,436,39] Disentangling with Adversarial Training When multiple styles or prosody information are entangled together, it is necessary to disentangle them during training for better expressive speech synthesis and control.…”

Section: Disentangling Controlling and Transferringmentioning

confidence: 99%

A Survey on Neural Speech Synthesis

Tan,

Qin,

Soong

et al. 2021

Preprint

View full text Add to dashboard Cite

Text to speech (TTS), or speech synthesis, which aims to synthesize intelligible and natural speech given text, is a hot research topic in speech, language, and machine learning communities and has broad applications in the industry. As the development of deep learning and artificial intelligence, neural network-based TTS has significantly improved the quality of synthesized speech in recent years. In this paper, we conduct a comprehensive survey on neural TTS, aiming to provide a good understanding of current research and future trends. We focus on the key components in neural TTS, including text analysis, acoustic models, and vocoders, and several advanced topics, including fast TTS, low-resource TTS, robust TTS, expressive TTS, and adaptive TTS, etc. We further summarize resources related to TTS (e.g., datasets, opensource implementations) and discuss future research directions. This survey can serve both academic researchers and industry practitioners working on TTS.

show abstract

Section: Disentangling Controlling and Transferringmentioning

confidence: 99%

A Survey on Neural Speech Synthesis

Tan,

Qin,

Soong

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, some efforts have been conducted to model the style in multiple scales or hierarchy [25], [41], [42]. In [41], the authors extended the GST to a hierarchical GST architecture, in which several GST layers are used with a residual connection, to learn hierarchical embedding information implicitly.…”

Section: B Reference Audio Based Expressive Speech Synthesismentioning

confidence: 99%

“…In their model, the first GST layer performs well at the speaker discrimination, while the representations from deeper GST layers tend to contain finer speaking styles or emotion variations. Different from this implicit way, in [25], a reference encoder-based model is trained explicitly to extract phonemelevel and global-level style features from mel-spectrograms. In this paper, the proposed method also learns information from different scales explicitly.…”

Section: B Reference Audio Based Expressive Speech Synthesismentioning

confidence: 99%

“…From the perspective of coarse granularity, we can summarize the overall style or emotion category of an entire utterance, for instance, we can say that the emotion conveyed by an utterance is "happy". From the view of fine granularity, each pronunciation unit in a sentence has its own characteristics such as intensity, speed, pitch, and energy [25]. In addition to the above granularities, some prosodic expressions like the fluctuations of intonation are also reflected in between-and cross-pronunciation unit relationships [26].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis

Long¹,

Yang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Expressive synthetic speech is essential for many human-computer interaction and audio broadcast scenarios, and thus synthesizing expressive speech has attracted much attention in recent years. Previous methods performed the expressive speech synthesis either with explicit labels or with a fixed-length style embedding extracted from reference audio, both of which can only learn an average style and thus ignores the multi-scale nature of speech prosody. In this paper, we propose MsEmoTTS, a multi-scale emotional speech synthesis framework, to model the emotion from different levels. Specifically, the proposed method is a typical attention-based sequence-to-sequence model and with proposed three modules, including global-level emotion presenting module (GM), utterance-level emotion presenting module (UM), and local-level emotion presenting module (LM), to model the global emotion category, utterance-level emotion variation, and syllable-level emotion strength, respectively. In addition to modeling the emotion from different levels, the proposed method also allows us to synthesize emotional speech in different ways, i.e., transferring the emotion from reference audio, predicting the emotion from input text, and controlling the emotion strength manually. Extensive experiments conducted on a Chinese emotional speech corpus demonstrate that the proposed method outperforms the compared reference audiobased and text-based emotional speech synthesis methods on the emotion transfer speech synthesis and text-based emotion prediction speech synthesis respectively. Besides, the experiments also show that the proposed method can control the emotion expressions flexibly. Detailed analysis shows the effectiveness of each module and the good design of the proposed method.

show abstract

“…To model and control local prosodic variations in speech, some previous works attempt to predict finer-grained speaking styles from text, such as word level [15] and phoneme level [16]. It is more widely accepted that the style expressions of human speech are multi-scale in nature [17,18], where the global-scale style is usually observed as emotion and the local-scale is more close to the prosody variation [19,20]. These styles from different levels work together to produce rich expressiveness in speech.…”

Section: Introductionmentioning

confidence: 99%

Towards Expressive Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis

Lei¹,

Zhou²,

Chen³

et al. 2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Previous works on expressive speech synthesis focus on modelling the mono-scale style embedding from the current sentence or context, but the multi-scale nature of speaking style in human speech is neglected. In this paper, we propose a multiscale speaking style modelling method to capture and predict multi-scale speaking style for improving the naturalness and expressiveness of synthetic speech. A multi-scale extractor is proposed to extract speaking style embeddings at three different levels from the ground-truth speech, and explicitly guide the training of a multi-scale style predictor based on hierarchical context information. Both objective and subjective evaluations on a Mandarin audiobooks dataset demonstrate that our proposed method can significantly improve the naturalness and expressiveness of the synthesized speech 1 .

show abstract

Towards Multi-Scale Style Control for Expressive Speech Synthesis

Cited by 4 publications

References 11 publications

A Survey on Neural Speech Synthesis

A Survey on Neural Speech Synthesis

MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis

Towards Expressive Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis

Contact Info

Product

Resources

About