Neural Speech Synthesis with Transformer Network

Abstract

Although end-to-end neural text-to-speech (TTS) methods (such as Tacotron2) are proposed and achieve state-of-theart performance, they still suffer from two problems: 1) low efficiency during training and inference; 2) hard to model long dependency using current recurrent neural networks (RNNs). Inspired by the success of Transformer network in neural machine translation (NMT), in this paper, we introduce and adapt the multi-head attention mechanism to replace the RNN structures and also the original attention mechanism in Tacotron2. With the help of multi-head self-attention, the hidden states in the encoder and decoder are constructed in parallel, which improves training efficiency. Meanwhile, any two inputs at different times are connected directly by a self-attention mechanism, which solves the long range dependency problem effectively. Using phoneme sequences as input, our Transformer TTS network generates mel spectrograms, followed by a WaveNet vocoder to output the final audio results. Experiments are conducted to test the efficiency and performance of our new network. For the efficiency, our Transformer TTS network can speed up the training about 4.25 times faster compared with Tacotron2. For the performance, rigorous human tests show that our proposed model achieves state-of-the-art performance (outperforms Tacotron2 with a gap of 0.048) and is very close to human quality (4.39 vs 4.44 in MOS).

A Comparative Study on Transformer vs RNN in Speech Applications

“…The input of the encoder in TTS is a sequence of IDs corresponding to the input characters and the EOS symbol. First, the character ID sequence is converted into a sequence of character vectors with an embedding layer, and then the positional encoding scaled by a learnable scalar parameter is added to the vectors [4]. This process is a TTS implementation of EncPre(·) in Eq.…”

“…This network consists of two linear layers with 256 units, a ReLU activation function, and dropout followed by a projection linear layer with d att units. Since it is expected that the hidden representations converted by Prenet are located in the similar feature space to that of encoder features, Prenet helps to learn a diagonal encoder-decoder attention [4]. Then the decoder DecBody(·) in Eq.…”

“…Currently, existing Transformer-based speech applications [2]- [4] still lack an open source toolkit and reproducible experiments while previous studies in NMT [5], [6] provide them. Therefore, we work on an open community-driven project for end-to-end speech applications using both Transformer and RNN by following the success of Kaldi for hidden Markov model (HMM)-based ASR [7].…”

Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram

“…In particular, the training process becomes 4.82 times faster (from 13.5 days to 2.8 days with two NVIDIA Telsa V100 GPUs) and the inference process becomes 1.96 times faster (from 14.62 to 28.68 real-time 2 to generate 24 kHz speech waveforms with a single NVIDIA Telsa V100 GPU) compared with the conventional ClariNet model. • We combined the proposed Parallel WaveGAN with a TTS acoustic model based on a Transformer [15][16][17]. The perceptual listening tests verify that the proposed Parallel WaveGAN achieves 4.16 MOS, which is competitive to the best distillation-based ClariNet model.…”

Unsupervised Style and Content Separation by Minimizing Mutual Information for Speech Synthesis

“…The original GST method uses an LSTM based Tacotron 2 [1] as the TTS backbone and an LSTM encoder for computing the style coefficients. For training efficiency and fair comparison, in our implementation of GST, we use Transformer TTS [3] for the content encoder and the decoder, and replace the LSTM with max-pooling for computing the style coefficients. We refer to our implementation of this method as GST * .…”