WaveTTS: Tacotron-based TTS with Joint Time-Frequency Domain Loss

Abstract

Tacotron-based text-to-speech (TTS) systems directly synthesize speech from text input. Such frameworks typically consist of a feature prediction network that maps character sequences to frequency-domain acoustic features, followed by a waveform reconstruction algorithm or a neural vocoder that generates the time-domain waveform from acoustic features. As the loss function is usually calculated only for frequency-domain acoustic features, that doesn't directly control the quality of the generated time-domain waveform. To address this problem, we propose a new training scheme for Tacotron-based TTS, referred to as WaveTTS, that has 2 loss functions: 1) timedomain loss, denoted as the waveform loss, that measures the distortion between the natural and generated waveform; and 2) frequency-domain loss, that measures the Mel-scale acoustic feature loss between the natural and generated acoustic features. WaveTTS ensures both the quality of the acoustic features and the resulting speech waveform. To our best knowledge, this is the first implementation of Tacotron with joint time-frequency domain loss. Experimental results show that the proposed framework outperforms the baselines and achieves high-quality synthesized speech.

An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning

“…As illustrated in Figure 8, encoder-decoder models with attention have recently shown considerable success in modeling a variety of complex sequence-to-sequence problems. Tacotron [87], [176], [208] represents one of the successful text-to-speech (TTS) implementations, that has been extended to voice conversion [3], [179].…”

Transforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Data

“…It plays an important role as a manifestation at semantic and pragmatic level of spoken languages. An adequate rendering of emotion in speech is critically important in expressive text-to-speech [2,3], personalized speech synthesis, and intelligent dialogue systems, such as social robots and conversational agents.…”