Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS

Abstract

Neural TTS has demonstrated strong capabilities to generate human-like speech with high quality and naturalness, while its generalization to out-of-domain texts is still a challenging task, with regard to the design of attention-based sequence-tosequence acoustic modeling. Various errors occur in those inputs with unseen context, including attention collapse, skipping, repeating, etc., which limits the broader applications. In this paper, we propose a novel stepwise monotonic attention method in sequence-to-sequence acoustic modeling to improve the robustness on out-of-domain inputs. The method utilizes the strict monotonic property in TTS with constraints on monotonic hard attention that the alignments between inputs and outputs sequence must be not only monotonic but allowing no skipping on inputs. Soft attention could be used to evade mismatch between training and inference. The experimental results show that the proposed method could achieve significant improvements in robustness on out-of-domain scenarios for phoneme-based models, without any regression on the in-domain naturalness test.

Location-Relative Attention Mechanisms for Robust Long-Form Speech Synthesis

“…The general form of this type of attention is shown in (5), where wi, Zi, ∆i, and σi are computed from the attention RNN state. The mean of each Gaussian component is computed using the recurrence relation in (6), which makes the mechanism location-relative and potentially monotonic if ∆i is constrained to be positive.…”

“…Approaches based on the seminal Tacotron system [3] have demonstrated naturalness that rivals that of human speech for certain domains [4]. Despite these successes, there are sometimes complaints of a lack of robustness in the alignment procedure that leads to missing or repeating words, incomplete synthesis, or an inability to generalize to longer utterances [5,6,7]. The original Tacotron system [3] used the content-based attention mechanism introduced in [2] to align the target text with the output spectrogram.…”

“…The hybrid mechanism still has occasional alignment issues which led a number of authors to develop attention mechanisms that directly exploit monotonicity [9,5,6]. These monotonic alignment mechanisms have demonstrated properties like increased alignment speed during training, improved stability, enhanced naturalness, and a virtual elimination of synthesis errors.…”

End-to-End Text-to-Speech Using Latent Duration Based on VQ-VAE

“…Many soft-attention mechanisms have been proposed to stabilize alignments to be monotonic (e.g. [5,6]). Hard monotonic alignment is an alternative alignment method to avoid fatal alignment errors [7].…”

Teacher-Student Training For Robust Tacotron-Based TTS

“…The use of scheduled sampling comes with negative effects that include misalignment between the natural speech frames and the predicted frames due to the fact that the temporal dependency of the acoustic sequence is disrupted. The techniques to improve out-ofdomain performance include the GAN-based TTS framework [28] that introduces both real and generated data sequences in discriminator training, and more recently, stepwise monotonic attention for neural TTS [8].…”