Transfer Learning From Speech Synthesis to Voice Conversion With Non-Parallel Training Data

Abstract

We present a novel voice conversion (VC) framework by learning from a text-to-speech (TTS) synthesis system, that is called TTS-VC transfer learning or TTL-VC for short. We first develop a multi-speaker speech synthesis system with sequence-to-sequence encoder-decoder architecture, where the encoder extracts the linguistic representations of input text, while the decoder, conditioned on target speaker embedding, takes the context vectors and the attention recurrent network cell output to generate target acoustic features. We take advantage of the fact that TTS system maps input text to speaker independent context vectors, thus re-purpose such a mapping to supervise the training of the latent representations of an encoder-decoder voice conversion system. In the voice conversion system, the encoder takes speech instead of text as the input, while the decoder is functionally similar to the TTS decoder. As we condition the decoder on a speaker embedding, the system can be trained on non-parallel data for any-to-any voice conversion. During voice conversion training, we present both text and speech to speech synthesis and voice conversion networks respectively. At run-time, the voice conversion network uses its own encoder-decoder architecture without the need of text input. Experiments show that the proposed TTL-VC system outperforms two competitive voice conversion baselines consistently, namely phonetic posteriorgram and AutoVC methods, in terms of speech quality, naturalness, and speaker similarity.

Voice adaptation by color-encoded frame matching as a multi-objective optimization problem for future games

“…The evaluations saw good results, although subjective, proved that classical methods such as only frequency warping has difficulty in competing with the likes of NN. Among contemporary research is [31] which proposes to utilize Text-to-Speech (TTS) as an approach to training a model via a Recurrent Neural Network (RNN) vocoder for transfer learning. Their results show improvement over simpler techniques, and a slight improvement over similar (TTS-utilizing) methods.…”

Emotional Voice Conversion: Theory, Databases and ESD

“…Studies also show that voice conversion benefits from the knowledge about linguistic content in the speech. For example, speaker voice conversion successfully leverages TTS [132,20,133] or ASR systems [134,135] that are phoneticallyinformed and trained on large speech corpus.…”

A Unique Glottal Flow Parameters based Features for Anti-spoofing Countermeasures in Automatic Speaker Verification

“…Furthermore, the scope of the research is also confined to LA attacks as synthetic speech production is becoming more accessible and capturing naturality. This is due to the fact that open source tools and datasets are available for researchers to explore leading to more versatile synthetic speech generators [5], [23], [34], [35].…”

“…Moreover, the countermeasures developed so far are less than a decade old and still have a scope of improvement in terms of reducing the False Acceptance ratios. Most of the research is based on specific type of attack [5], [6] while few others consider all the types of attack making them universal detectors [7], [8].…”