Bootstrapping Non-Parallel Voice Conversion from Speaker-Adaptive Text-to-Speech

Abstract

Voice conversion (VC) and text-to-speech (TTS) are two tasks that share a similar objective, generating speech with a target voice. However, they are usually developed independently under vastly different frameworks. In this paper, we propose a methodology to bootstrap a VC system from a pretrained speaker-adaptive TTS model and unify the techniques as well as the interpretations of these two tasks. Moreover by offloading the heavy data demand to the training stage of the TTS model, our VC system can be built using a small amount of target speaker speech data. It also opens up the possibility of using speech in a foreign unseen language to build the system. Our subjective evaluations show that the proposed framework is able to not only achieve competitive performance in the standard intra-language scenario but also adapt and convert using speech utterances in an unseen language.

Pretraining Techniques for Sequence-to-Sequence Voice Conversion

“…Another line of work show that training an integrated system capable of performing either TTS or VC can boost individual performances [29], [54]. Bootstrapping from a speaker-adaptive or multispeaker TTS model is another active direction [55], [56]. However, few of the abovementioned methods were designed to tackle the data deficiency problem for seq2seq based parallel, one-to-one VC, which is the main scope of this paper.…”

Transfer Learning From Speech Synthesis to Voice Conversion With Non-Parallel Training Data

“…The text transcripts are only required during training. Zhang et al [46], and Luong et al [45] proposed joint training of TTS and VC by sharing a common decoder. Park et al [44] proposed to use the context vectors in Tacotron system as speaker-independent linguistic representation to guide the voice conversion.…”

“…The VC encoder seeks to generate speakerindependent linguistic features from input spectral features, while the VC decoder reconstructs the mel-spectrum features from the linguistic features, conditioning on a speaker code. Studies show that voice conversion benefits from explicit phonetic modeling that ensure adherence to linguistic content during conversion [44], [45].…”

“…The three competing baselines in Table II include a multispeaker Park et al [44] no non-parallel text and speech encoder and decoder yes any-to-many Luong et al [45] no non-parallel speech decoder yes any-to-one Zhang et al [46] no parallel text and speech decoder yes many-to-one Huang et al [47] no parallel speech decoder yes one-to-one TTL-VC yes non-parallel speech decoder no any-to-any…”

“…Park et al [44] proposed a transcription-guided speech encoder as part of a sequence-to-sequence TTS model for any-to-many voice conversion, which requires both text and speech as input during run-time inference. On the other hand, Luong et al [45] proposed to bootstrap a voice conversion system from a pre-trained speaker-adaptive TTS model, where both voice conversion and TTS share a common decoder. This method only handles feature mapping yet leaves the alignment task to an external system.…”