Sample Efficient Adaptive Text-to-Speech

Abstract

We present a meta-learning approach for adaptive text-to-speech (TTS) with few data. During training, we learn a multi-speaker model using a shared conditional WaveNet core and independent learned embeddings for each speaker. The aim of training is not to produce a neural network with fixed weights, which is then deployed as a TTS system. Instead, the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers. We introduce and benchmark three strategies: (i) learning the speaker embedding while keeping the WaveNet core fixed, (ii) fine-tuning the entire architecture with stochastic gradient descent, and (iii) predicting the speaker embedding with a trained neural network encoder. The experiments show that these approaches are successful at adapting the multi-speaker neural network to new speakers, obtaining state-of-the-art results in both sample naturalness and voice similarity with merely a few minutes of audio data from new speakers.

Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech

“…SEA-TTS [9] uses Wavenet [1] as its basic architecture and compares the two voice cloning approaches. As mentioned by [8] that adapting the whole model might result in overfitting, SEA-TTS proposed two techniques to deal with it.…”

“…There are two general approaches to deal with such task [8], speaker adaptation [8], [9], [10], [11], [12] and speaker encoding [6], [13], [7], [14], [15], [16], [17]. The speaker encoding method builds a multi-speaker TTS architecture which consists of a speaker encoder and a TTS model.…”

“…The speaker encoding method builds a multi-speaker TTS architecture which consists of a speaker encoder and a TTS model. The speaker encoder could be pre-trained [6] or jointly trained [8], [9] with the TTS model. In order to clone the voice of an unseen speaker, the speaker encoder extracts the speaker's embedding from a few speech samples.…”

“…However, the speaker encoder might suffer from generalization problems and adapt worse for unseen speakers. On the other hand, the multi-speaker TTS architecture includes a speaker embedding table instead of a speaker encoder for the speaker adaptation method [8], [9], [11]. The speaker embedding table is jointly trained with the TTS model.…”

“…When learning an unseen speaker's voice, we would first randomly initialize an embedding for the new speaker, and then the embedding would be fine-tuned by the speech samples of the unseen speaker alone or with the TTS model. As experimented in [8], [9], although the speaker adaptation approach performs better than the speaker encoding approach, it requires thousands of adaptation steps, which means more cloning time and computational resources are needed for high-quality voice cloning.…”