Semi-Supervised Speaker Adaptation for End-to-End Speech Synthesis with Pretrained Models

Inoue, Katsuki; Hara, Sunao; Abe, Masanobu; Hayashi, Tomoki; Yamamoto, Ryōichi; Watanabe, Shinji

doi:10.1109/icassp40776.2020.9053371

Cited by 16 publications

(10 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Theoretically speaking, as speaker characteristics are selfcontained within an utterance we should be able to clone voices without using text. One practical approach is obtaining automatically annotated transcriptions using a SOTA ASR system [16]. However ASR-predicted transcriptions contain wrong annotations, which affects the performance of the adaptation.…”

Section: B Training Voice Conversion System For Target Speakermentioning

confidence: 99%

“…When cloning voices, we fine-tuned both acoustic and vocoder models with the transcribed speech of the targets. This system represented a simple supervised approach by fine-tuning a well-trained single speaker model [16]. • VCM u : our unsupervised VC system followed the adaptation process described in Sec.…”

Section: B Capturing Unique Speaker Characteristicsmentioning

confidence: 99%

“…While there are many different approaches proposed to tackle this problem, they all share the same fundamental principle which is using an abundant corpus to compensate for the lack of data of a target speaker [15]. For neural TTS, we can fine-tune all or part of a well-trained acoustic model using transcribed speech from a target speaker [16]. For neural VC, we can pool the speech data of multiple source and target speakers and share knowledge learned from each [17].…”

Section: Introductionmentioning

confidence: 99%

“…Disentangling speaker characteristics from linguistic information and representing it as a speaker vector is hence a popular way for cloning voices [18]. Another approach is to use labels auto-generated by speakerindependent automatic speech recognition (ASR) trained on large-scale multi-speaker corpora [16]. Either way, the cloning method is usually formulated for a specific data scenario of a specific speech generation system (either TTS or VC), while a true data-efficient method should work on extremely limited data and also abundant data with or without labels.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

NAUTILUS: A Versatile Voice Cloning System

Luong

Yamagishi

2020

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

We introduce a novel speech synthesis system, called NAUTILUS, that can generate speech with a target voice either from a text input or a reference utterance of an arbitrary source speaker. By using a multi-speaker speech corpus to train all requisite encoders and decoders in the initial training stage, our system can clone unseen voices using untranscribed speech of target speakers on the basis of the backpropagation algorithm. Moreover, depending on the data circumstance of the target speaker, the cloning strategy can be adjusted to take advantage of additional data and modify the behaviors of text-to-speech (TTS) and/or voice conversion (VC) systems to accommodate the situation. We test the performance of the proposed framework by using deep convolution layers to model the encoders, decoders and WaveNet vocoder. Evaluations show that it achieves comparable quality with state-of-the-art TTS and VC systems when cloning with just five minutes of untranscribed speech. Moreover, it is demonstrated that the proposed framework has the ability to switch between TTS and VC with high speaker consistency, which will be useful for many applications.

show abstract

Section: B Training Voice Conversion System For Target Speakermentioning

confidence: 99%

Section: B Capturing Unique Speaker Characteristicsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

NAUTILUS: A Versatile Voice Cloning System

Luong

Yamagishi

2020

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

show abstract

“…[15] presents speaker, language and stress/tone embeddings used for TTS that can synthesize speech in multiple speaker identities and languages. Other techniques employ transfer learning methods to benefit from other speech processing applications, such as speaker verification [16] or speech recognition [17].…”

Section: Introductionmentioning

confidence: 99%

Speaker verification-derived loss and data augmentation for DNN-based multispeaker speech synthesis

Lorincz¹,

Stan²,

Giurgiu³

2021

Preprint

View full text Add to dashboard Cite

Building multispeaker neural network-based textto-speech synthesis systems commonly relies on the availability of large amounts of high quality recordings from each speaker and conditioning the training process on the speaker's identity or on a learned representation of it. However, when little data is available from each speaker, or the number of speakers is limited, the multispeaker TTS can be hard to train and will result in poor speaker similarity and naturalness.In order to address this issue, we explore two directions: forcing the network to learn a better speaker identity representation by appending an additional loss term; and augmenting the input data pertaining to each speaker using waveform manipulation methods. We show that both methods are efficient when evaluated with both objective and subjective measures. The additional loss term aids the speaker similarity, while the data augmentation improves the intelligibility of the multispeaker TTS system.

show abstract

Data-Efficient TTS

Tan

2023

Artificial Intelligence: Foundations, Theory, and Algorithms

View full text Add to dashboard Cite

Semi-Supervised Speaker Adaptation for End-to-End Speech Synthesis with Pretrained Models

Cited by 16 publications

References 21 publications

NAUTILUS: A Versatile Voice Cloning System

NAUTILUS: A Versatile Voice Cloning System

Speaker verification-derived loss and data augmentation for DNN-based multispeaker speech synthesis

Data-Efficient TTS

Contact Info

Product

Resources

About