Pre-Alignment Guided Attention for Improving Training Efficiency and Model Stability in End-to-End Speech Synthesis

Abstract

Recently, end-to-end (E2E) neural text-to-speech systems, such as Tacotron2, have begun to surpass the traditional multi-stage hand-engineered systems, with both simplified system building pipelines and high-quality speech. With a unique encoder-decoder neural structure, the Tacotron2 system no longer needs separately learned text analysis front-end, duration model, acoustic model, and audio synthesis module. The key of such a system lies in the attention mechanism, which learns an alignment between the encoder and the decoder, serving as an implicit duration model bridging the text sequence and the acoustic sequence. However, attention learning suffers from low training efficiency and model instability problems, which hinder the E2E approaches from wide deployment. In this paper, we address the problems and propose a novel pre-alignment guided attention learning approach. Specifically, we inject handy prior knowledge-accurate phoneme durations-in the neural network loss function to bias the attention learning to the desired direction more accurately. The explicit time alignment between an audio recording and its corresponding phoneme sequence can be achieved by forced-alignment from an automatic speech recognizer (ASR). The experiments show that the proposed pre-alignment guided (PAG) attention approach can significantly improve training efficiency and model stability. More specifically, the PAG updated version of Tacotron2 can quickly obtain the attention alignment using only 500 text, audio pairs, which is apparently not possible for the original Tacotron2. A series of subjective experiments also show that the PAG-Tacotron2 approach can synthesize more stable and natural speech.

Investigation of Fast and Efficient Methods for Multi-Speaker Modeling and Speaker Adaptation

“…1. Both GA and Force-Align model perform much better than the baseline, which is consistent with that in [18] and [19]. And MAGT model receives the most preferences compared to other models, which indicates the advantages of the multi-alignment guided attention.…”

“…Specifically, two extra alignment terms are introduced to penalize the attention learning if the learned alignment does not match with the guided alignments. Formally, we define two matrices A * , A to represent corresponding two guided alignments, where A * is achieved beforehand by force-aligning using ASR [19], A it's created with the assumption that the alignment between the input text sequence and output sequence should be "nearly diagonal" [18]. That is, A is defined as follows:…”

“…The first one is to use a constraint on the attention matrix to promote it to be "nearly diagonal" [18], which could be seen as providing a soft label for the alignments. Another one is to use the force-alignment results from an automatic speech recognition (ASR) module to penalize the attention learning [19], which offers a hard label for the alignments. Inspired by the successful use of interpolating the "hard label" with a "soft label" for each frame in domain ASR adaptation [20], thus instead of conditionally choosing from either the soft or hard label, we further combine these two methods to take full advantages of them for faster and efficient speaker adaptation.…”

RobuTrans: A Robust Transformer-Based Text-to-Speech Model

“…We conduct experiments investigating the robustness of other attention mechanisms, including forward attention (Zhang, Ling, and Dai 2018), GMM attention (Graves 2013), forced monotonic mechanism (Raffel et al 2017) and guided attention (Zhu et al 2019). All these mechanisms generate bad cases, therefore none of them could be part of our robust model.…”

DNAE-GAN: Noise-free acoustic signal generator by integrating autoencoder and generative adversarial network

“…Therefore, even if the same sentence is continuously repeated during the same period of time, it is impossible for the collected audio signal to be completely similar every time. However, in related applications to robots, most employ the fixed text-to-speech (TTS) mode 26,27 technology to enable users to understand robots’ expressions by listening. Although TTS technology is increasingly maturing, and users can easily understand its content, the audio signals generated by TTS usually exhibit relatively flat intonation and generate the same audio signals each time they are played.…”