Contrastive Predictive Coding Supported Factorized Variational Autoencoder for Unsupervised Learning of Disentangled Speech Representations

Abstract

In this work we tackle disentanglement of speaker and content related variations in speech signals. We propose a fully convolutional variational autoencoder employing two encoders: a content encoder and a speaker encoder. To foster disentanglement we propose adversarial contrastive predictive coding. This new disentanglement method does neither need parallel data nor any supervision, not even speaker labels. With successful disentanglement the model is able to perform voice conversion by recombining content and speaker attributes. Due to the speaker encoder which learns to extract speaker traits from an audio signal, the proposed model not only provides meaningful speaker embeddings but is also able to perform zero-shot voice conversion, i.e. with previously unseen source and target speakers. Compared to state-of-the-art disentanglement approaches we show competitive disentanglement and voice conversion performance for speakers seen during training and superior performance for unseen speakers.

Voice Conversion Based Speaker Normalization for Acoustic Unit Discovery

“…The proposed system is shown in Figure 1. It consists of an adversarial CPC based VC system [22] for speaker normalization and a subsequent HMMVAE [7] to perform the AUD.…”

“…For the voice conversion we here employ a Factorized Variational Autoencoder (FVAE) along with adversarial CPC as proposed in [22], which has shown to yield a well-balanced trade-off between linguistic content preservation and speaker invariance. The FVAE employs two convolutional encoders, namely, a content encoder outputting a series of content embeddings C = (c1, .…”

“…For the VC system, the parameters are: A Blackman window with a window length of 400 samples, a window shift of 160 samples, an FFT size of 512 samples and 80 filters in the mel filterbank. These parameters are taken from [22]. Each of the mel-bands is normalized to zero mean and unit variance, where normalization statistics are computed using the complete training set.…”

Learning Disentangled Phone and Speaker Representations in a Semi-Supervised VQ-VAE Paradigm

“…Recent work from [10] aims to disentangle content from speaker for voice conversion tasks. While their technique does not necessarily rely on speaker labels, it does use vocal tract length perturbation (VTLP) to facilitate regularization during adversarial disentanglement.…”

CVC: Contrastive Learning for Non-parallel Voice Conversion

“…In more recently emerged works, VAE has been combined with contrastive predictive coding (CPC) [12] to tackle voice conversion. In [13], CPC is used as an additional regularization objective for enhancing the content encoder. VQVAE-CPC [14] used vector-quantized VAE (VQ-VAE) [15] where the quantized latent representations could discard undesired attributes from the source speech more effectively.…”