Unsupervised Learning of a Disentangled Speech Representation for Voice Conversion

Abstract

This paper presents an approach to voice conversion, which does neither require parallel data nor speaker or phone labels for training. It can convert between speakers which are not in the training set by employing the previously proposed concept of a factorized hierarchical variational autoencoder. Here, linguistic and speaker induced variations are separated upon the notion that content induced variations change at a much shorter time scale, i.e., at the segment level, than speaker induced variations, which vary at the longer utterance level. In this contribution we propose to employ convolutional instead of recurrent network layers in the encoder and decoder blocks, which is shown to achieve better phone recognition accuracy on the latent segment variables at frame-level due to their better temporal resolution. For voice conversion the mean of the utterance variables is replaced with the respective estimated mean of the target speaker. The resulting log-mel spectra of the decoder output are used as local conditions of a WaveNet which is utilized for synthesis of the speech waveforms. Experiments show both good disentanglement properties of the latent space variables, and good voice conversion performance.

Integration of variational autoencoder and spatial clustering for adaptive multi-channel neural speech separation

“…Among the different models rising in the field in deep generative modeling, variational auto-encoder [17,18] has become probably the most popular in speech processing [19,20]. Variational auto-encoder is a deep latent variable model, where the probability density of the observation depends on the output of a highly non-linear transformation of the latent variable, where the transformation is modeled by a neural network (known as a decoder).…”

“…In our work, we use VAE to model single-speaker speech in the log-magnitude STFT domain. We use a (slightly modified) model from [20], which was inspired by [19]. In this model, part of the latent variables is trained to model speaker variability.…”

“…Note that the speaker classification objective differs from [19,20], where the aim was to learn to disentangle the speaker information in an unsupervised way. Here, we assume having the speaker identities during training, and we can thus directly use a supervised objective.…”

“…The value inside the expectation expresses how likely the source k is dominant and has the value of the observed mixed speech y t,f . For example of k = 1, this can be expressed as ln p(y t,f , d t,f = 1|Z (1) , Z (2) (20) where Φ(•) is the cumulative normal distribution. This solution can be also found in previous literature on lifted-max model [7,11].…”

“…To combine the strengths of both stated directions, we here propose to use a neural network as a generative model of the speech signal in a factorial model. For this, we use the variational autoencoder (VAE) [17,18], which has been previously successfully used to model speech [19,20]. We train VAE on clean single-speaker speech and create a simple Gaussian model of noise.…”