Fine-grained style control in Transformer-based Text-to-speech Synthesis

Abstract

In this paper, we present a novel architecture to realize fine-grained style control on the transformer-based text-tospeech synthesis (TransformerTTS). Specifically, we model the speaking style by extracting a time sequence of local style tokens (LST) from the reference speech. The existing content encoder in TransformerTTS is then replaced by our designed cross-attention blocks for fusion and alignment between content and style. As the fusion is performed along with the skip connection, our cross-attention block provides a good inductive bias to gradually infuse the phoneme representation with a given style. Additionally, we prevent the style embedding from encoding linguistic content by randomly truncating LST during training and using wav2vec 2.0 features. Experiments show that with fine-grained style control, our system performs better in terms of naturalness, intelligibility, and style transferability. Our code and samples are publicly available. 1

Controllable speech synthesis by learning discrete phoneme-level prosodic representations

“…[13] shows that discretizing phoneme-level latent features and using an autoregressive prior generates more natural samples instead of simply sampling from a standard VAE prior. Extending the idea of GSTs, a pretrained wav2vec 2.0 [14] model can be used to capture local style patterns in a transformer-based architecture [15]. In [16], prosody is controlled by incorporating a word-level GST module in a non-attentive Tacotron model [17], with the addition of an autoregressive prior which allows high quality speech synthesis without requiring a reference audio.…”

GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech

“…4) FG-TransformerTTS(Chen & Rudnicky, 2021): The finegrained style control on auto-regressive model Transformer-TTS. 5) Expressive FastSpeech 2(Ren et al, 2020): The combination of both multi-speaker(Chen et al, 2020b) and muli-emotion(Cui et al, 2021) FastSpeech 2, which adds the speaker and emotion d-vectors extracted by the pretrained discriminative models to the backbone.…”