Prosodic Clustering for Phoneme-Level Prosody Control in End-to-End Speech Synthesis

Abstract

This paper presents a method for controlling the prosody at the phoneme level in an autoregressive attention-based text-to-speech system. Instead of learning latent prosodic features with a variational framework as is commonly done, we directly extract phoneme-level F0 and duration features from the speech data in the training set. Each prosodic feature is discretized using unsupervised clustering in order to produce a sequence of prosodic labels for each utterance. This sequence is used in parallel to the phoneme sequence in order to condition the decoder with the utilization of a prosodic encoder and a corresponding attention module. Experimental results show that the proposed method retains the high quality of generated speech, while allowing phoneme-level control of F0 and duration. By replacing the F0 cluster centroids with musical notes, the model can also provide control over the note and octave within the range of the speaker.

Controllable speech synthesis by learning discrete phoneme-level prosodic representations

“…For the duration feature, clustering is performed separately per phoneme as phoneme classes, such as vowels and consonants, differ substantially depending on their articulation characteristics. Results from our previous work [46] show that voice quality deterioration when using the outermost clusters is not so severe in F0 control compared to duration control. Thus, we have adopted a balanced clustering method for extracting duration clusters.…”

“…Instead of training a quantized fine-grained VAE in order to learn latent representations, we simply use extracted features such as F0 and duration, the values of which are determined by standard speech processing tools. The discretization is then performed at the phoneme level using simple clustering methods, such as K-Means clustering, resulting in humanly interpretable labels which are directly applied to the dataset without requiring training [46]. An additional group of encoder and attention modules learn to model the discrete sequences and disentangle their content from the corresponding phoneme-level linguistic features.…”

“…For F0 values, we apply per speaker normalization and K-Means clustering with the purpose of neutralizing speaker and gender variations and obtain speaker-independent labels which explain the same space. A balanced duration clustering strategy is used for the durations of each phoneme separately, assigning an equal number of samples in each cluster in order to address instabilities due to cluster imbalance, a problem which was found in our previous work [46] and we would like to alleviate. Augmentation transformations are also applied to the training data [48], in order to increase the number of samples in the outermost clusters.…”

Unsupervised word-level prosody tagging for controllable speech synthesis

“…They disentangle these prosody features and provide more independent control. [9,10] model these features with clustering, which is a purely data-driven method that have more interpretability. In contrast to explicit representation, implicit prosody representation is more complete and richer when modelling prosody diversity, yet uninterpretable.…”

A Study of Modeling Rising Intonation in Cantonese Neural Speech Synthesis

“…In order to improve the generated prosody, several variational [7,8] and non-variational [9,10] methods have been proposed to learn latent prosodic representations. Some methods [11,12] are proposed for low-level prosody control. In inference, though some prosody attributes like emotions can be captured and transferred through reference audios, other prosody attributes related to the context could be inappropriate.…”