Speaker and Expression Factorization for Audiobook Data: Expressiveness and Transplantation

Abstract

Expressive synthesis from text is a challenging problem. There are two issues. First, read text is often highly expressive to convey the emotion and scenario in the text. Second, since the expressive training speech is not always available for different speakers, it is necessary to develop methods to share the expressive information over speakers. This paper investigates the approach of using very expressive, highly diverse audiobook data from multiple speakers to build an expressive speech synthesis system. Both of two problems are addressed by considering a factorized framework where speaker and emotion are modeled in separate sub-spaces of a cluster adaptive training (CAT) parametric speech synthesis system. The sub-spaces for the expressive state of a speaker and the characteristics of the speaker are jointly trained using a set of audiobooks. In this work, the expressive speech synthesis system works in two distinct modes. In the first mode, the expressive information is given by audio data and the adaptation method is used to extract the expressive information in the audio data. In the second mode, the input of the synthesis system is plain text and a full expressive synthesis system is examined where the expressive state is predicted from the text. In both modes, the expressive information is shared and transplanted over different speakers. Experimental results show that in both modes, the expressive speech synthesis method proposed in this work significantly improves the expressiveness of the synthetic speech for different speakers. Finally, this paper also examines whether it is possible to predict the expressive states from text for multiple speakers using a single model, or whether the prediction process needs to be speaker specific.Index Terms-Audiobook, cluster adaptive training, expressive speech synthesis, factorization, hidden Markov model, neural network.

Speech Synthesis for the Generation of Artificial Personality

“…In the case of the unit selection synthesis, there are 6 different versions, 3 female (neutral, lax and tense) and 3 male (neutral, lax and tense). considering how to effectively model voice quality within parametric speech synthesis, e.g., [48], [49], [50], [51], [52], however this is atypical and in the work presented here the parametric system that is evaluated is very close to the design described in Zen et al [22]. The approach does not directly model voice quality, but includes mixed excitation where noise representing frication is added to a pulse train to generate speech.…”

Can We Generate Emotional Pronunciations for Expressive Speech Synthesis?

“…This solution requires as many speech databases as required speaking styles and raises the issue of the consistency between semantics and expressivity. A speaker and expressivity factorization could help to solve this problem [21]. Otherwise, expressivity can also be controlled in symbolic terms (diphone identity, position, etc.)…”

Effective Emotion Transplantation in an End-to-End Text-to-Speech System

“…Because of the success of end-to-end text-to-speech (E2E-TTS) models, researchers have been trying to expand this framework to synthesize more expressive speech [9]- [13]. Unlike emotionally neutral speech (narrative speech) which has monotonic prosody, expressive speech has many variations in prosody.…”