Crank: An Open-Source Software for Nonparallel Voice Conversion Based on Vector-Quantized Variational Autoencoder

Abstract

In this paper, we present an open-source software for developing a nonparallel voice conversion (VC) system named crank. Although we have released an open-source VC software based on the Gaussian mixture model named sprocket in the last VC Challenge, it is not straightforward to apply any speech corpus because it is necessary to prepare parallel utterances of source and target speakers to model a statistical conversion function. To address this issue, in this study, we developed a new open-source VC software that enables users to model the conversion function by using only a nonparallel speech corpus. For implementing the VC software, we used a vector-quantized variational autoencoder (VQVAE). To rapidly examine the effectiveness of recent technologies developed in this research field, crank also supports several representative works for autoencoder-based VC methods such as the use of hierarchical architectures, cyclic architectures, generative adversarial networks, speaker adversarial training, and neural vocoders. Moreover, it is possible to automatically estimate objective measures such as mel-cepstrum distortion and pseudo mean opinion score based on MOSNet. In this paper, we describe representative functions developed in crank and make brief comparisons by objective evaluations.

Non-Parallel Voice Conversion for ASR Augmentation

“…PPGs acts as speaker-invariant representations that can be easily utilized to achieve conversion. The other main approach is with auto-encoding style training approaches, utilizing both variational auto-encoder (VAE) [3,4,5,6,7,8] and vector-quantized VAE (VQ-VAE) [9,10,11,12] approaches. The auto-encoding style training approaches try to disentangle speaker speaker information from the content of source speech, using various methods from bottlenecking to adversarial training to prevent speaker leakage in the content or speech encoder.…”

A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion

“…For the VAE model, we used crank [15], an open-source VC software that combines recent advances in autoencoder-based VC methods, including the use of hierarchical architectures, cyclic loss and adversarial training. To take full advantage of unsupervised learning, we trained the network using not only the data of the patient and the reference speakers but also a multi-speaker TTS dataset.…”

“…The ability of seq2seq VC models to convert suprasegmental information and the parallel training strategy can greatly improve the naturalness and intelligibility, though the speaker identity is changed into that of the reference speaker. Next, a frame-wise, nonparallel VC model realized by a variational autoencoder (VAE) [13,14,15] takes the converted speech with the identity of the reference speaker as input and restores the identity of the patient. An important assumption we make here is that due to the frame-wise constraint, the VAE model changes only time-invariant characteristics such as the speaker identity, while preserving time-variant characteristics, such as pronunciation.…”

TDASS: Target Domain Adaptation Speech Synthesis Framework for Multi-speaker Low-Resource TTS

“…Previous researches on multi-speaker TTS remain requires hundreds or thousands of high-quality training speeches per person [15]- [17]. Several studies utilize voice conversion [18]- [20] to augment both speaker and speech databases to address the extensive training data required. But the recent advance of voice conversion is hard to synthesize noisefree speeches.…”