VQVAE Unsupervised Unit Discovery and Multi-Scale Code2Spec Inverter for Zerospeech Challenge 2019

Abstract

We describe our submitted system for the ZeroSpeech Challenge 2019. The current challenge theme addresses the difficulty of constructing a speech synthesizer without any text or phonetic labels and requires a system that can (1) discover subword units in an unsupervised way, and (2) synthesize the speech with a target speaker's voice. Moreover, the system should also balance the discrimination score ABX, the bit-rate compression rate, and the naturalness and the intelligibility of the constructed voice. To tackle these problems and achieve the best tradeoff, we utilize a vector quantized variational autoencoder (VQ-VAE) and a multi-scale codebook-tospectrogram (Code2Spec) inverter trained by mean square error and adversarial loss. The VQ-VAE extracts the speech to a latent space, forces itself to map it into the nearest codebook and produces compressed representation. Next, the inverter generates a magnitude spectrogram to the target voice, given the codebook vectors from VQ-VAE. In our experiments, we also investigated several other clustering algorithms, including K-Means and GMM, and compared them with the VQ-VAE result on ABX scores and bit rates. Our proposed approach significantly improved the intelligibility (in CER), the MOS, and discrimination ABX scores compared to the official ZeroSpeech 2019 baseline or even the topline.

Speech-to-Speech Translation Between Untranscribed Unknown Languages

“…Since the speech utterances for the sentences are unavailable, we generated sentences with Google text-to-speech API for all languages pairs. Even though the lack of natural speech dataset in this paper, VQ-VAE and codebook inverter can be applied and has shown a great performance on multispeaker natural speech [14,13]. Some papers [30,31,32] also show the performance improvement from the synthetic dataset can be carried over to the real dataset.…”

Tackling Perception Bias in Unsupervised Phoneme Discovery Using DPGMM-RNN Hybrid Model and Functional Load

“…Machines can directly get discrete segments by applying such clustering algorithms as K-means [12], [11], GMM [11], or DPGMM clustering [13], [1], [14] from the acoustic features. The DPGMM algorithm [28] retained the state-ofthe-art approach in the Zerospeech 2015 and 2017 [15], [16].…”

“…However, low-dimensional continuous features are never as efficient as discrete features or discrete segments. The Vector Quantised-Variational AutoEncoder (VQ-VAE) can quantize speech acoustic features [11].…”

“…Recently the Zero Resource Speech Challenge [6] was organized to compare the performance of these methods. Typical methods include neural network technology, such as representation learning by autoencoder [7], [8], [9] or discriminative training by ABnet [10], traditional clustering such as GMM [11] or k-means [12], [11], and nonparametric clustering such as the Dirichlet Process Gaussian Mixture Model (DPGMM) trained by Gibbs sampling [1], or variational inference [13], [14]. Among them, DPGMM, which is acoustic clustering, achieved the top performance at Zerospeech 2015 and 2017 [15], [16].…”

Bootstrapping Non-Parallel Voice Conversion from Speaker-Adaptive Text-to-Speech

“…For the conventional Gaussian mixture model (GMM) approach, non-parallel VC can be adapted from a pretrained parallel VC in the model space using the maximum a posterior (MAP) method [7,8] or as interpolation between multiple parallel models [9]. For recent neural network approaches, a non-parallel VC can be trained by directly using an intermediate linguistic representation extracted from an automatic speech recognition (ASR) model [10,11] or by indirectly encouraging the network to disentangle linguistic information from the speaker characteristics using methods like variational autoencoder (VAE) [12], generative adversarial networks (GAN) [13,14] or some other techniques [15]. For both parallel and non-parallel VC, the systems usually change the voice but are unable to change the duration of the utterance.…”