A Vector Quantized Variational Autoencoder (VQ-VAE) Autoregressive Neural $F_0$ Model for Statistical Parametric Speech Synthesis

Abstract

Recurrent neural networks (RNNs) can predict fundamental frequency (F0) for statistical parametric speech synthesis systems, given linguistic features as input. However, these models assume conditional independence between consecutive F0 values, given the RNN state. In a previous study, we proposed autoregressive (AR) neural F0 models to capture the causal dependency of successive F0 values. In subjective evaluations, a deep AR model (DAR) outperformed an RNN. Here, we propose a Vector Quantized Variational Autoencoder (VQ-VAE) neural F0 model that is both more efficient and more interpretable than the DAR. This model has two stages: one uses the VQ-VAE framework to learn a latent code for the F0 contour of each linguistic unit, and other learns to map from linguistic features to latent codes. In contrast to the DAR and RNN, which process the input linguistic features frame-by-frame, the new model converts one linguistic feature vector into one latent code for each linguistic unit. The new model achieves better objective scores than the DAR, has a smaller memory footprint and is computationally faster. Visualization of the latent codes for phones and moras reveals that each latent code represents an F0 shape for a linguistic unit.

Camp: A Two-Stage Approach to Modelling Prosody in Context

“…Most prosody representations are learnt at the sentence-level [16,17]. However, these are too coarse and are not able to perfectly reconstruct prosody [18]. To accurately capture prosody we need a sequence of representations, e.g.…”

“…To accurately capture prosody we need a sequence of representations, e.g. word or phrase level [18,19]. Following CopyCat [20], we learn a sequence of representations from the mel-spectrogram, but we do so at the word-level (stage-1).…”

“…Reducing the bit rate of the representation, by using a longer unit length, should allow the learnt prior to capture longer-range effects, such as prosodic patterns. The linguistic linker introduced by Wang et al [18] experiments with different unit lengths when modelling F 0 , but not other aspects of prosody such as rhythm and intensity. Representations extracted from a spectrogram can capture these aspects of prosody [21,17].…”

End-to-End Text-to-Speech Using Latent Duration Based on VQ-VAE

“…VQ-VAE [14] has been applied to various speech synthesis tasks, including diverse and controllable TTS [17,18], a new TTS framework based on symbol-to-symbol translation [19], speech coding [20], voice conversion [21], and representation learning [22,23,24,25].…”

Semi-Supervised Adversarial Variational Autoencoder

“…The VAEs have met with great success in recent years in several applicative areas including anomaly detection [6][7][8][9], text classification [10], sentence generation [11], speech synthesis and recognition [12][13][14], spatio-temporal solar irradiance forecasting [15] and in geoscience for data assimilation [2]. In other respects, the two major application areas of the VAEs are the biomedical and healthcare recommendation [16][17][18][19], and industrial applications for nonlinear processes monitoring [1,3,4,[20][21][22][23][24][25].…”