End-to-End Text-to-Speech for Low-Resource Languages by Cross-Lingual Transfer Learning

Abstract

End-to-end text-to-speech (TTS) has shown great success on large quantities of paired text plus speech data. However, laborious data collection remains difficult for at least 95% of the languages over the world, which hinders the development of TTS in different languages. In this paper, we aim to build TTS systems for such low-resource (target) languages where only very limited paired data are available. We show such TTS can be effectively constructed by transferring knowledge from a high-resource (source) language. Since the model trained on source language cannot be directly applied to target language due to input space mismatch, we propose a method to learn a mapping between source and target linguistic symbols. Benefiting from this learned mapping, pronunciation information can be preserved throughout the transferring procedure. Preliminary experiments show that we only need around 15 minutes of paired data to obtain a relatively good TTS system. Furthermore, analytic studies demonstrated that the automatically discovered mapping correlate well with the phonetic expertise.

Hierarchical Transfer Learning for Multilingual, Multi-Speaker, and Style Transfer DNN-Based TTS on Low-Resource Languages

“…ASR-TTS proposed by [30] need additional ASR to assist TTS learning. Mapping-based DTL is explored in [31] by adding a phonetic transformation network (PTN) model to learn a mapping between source and target linguistic symbols. An ASR system is used to train PTN separately.…”

“…Our strategy is simpler than [30] as it does not need additional system such as ASR. Similar to [31], we apply DTL approach. However, our DTL is network-based approach that is more flexible than mapping-based DTL applied by [31] in which with multi stages of transfer learning the previous learned DNN parameters can be passed on to a larger network.…”

“…Speech chain machine for cross-lingual is proposed by [30] that applies cycle consistency training for cross-lingual ASR-TTS. Work [31] proposed an approach to discover cross-lingual symbol mapping from abundant source data.…”

“…Especially for deep learning, a study by [26] classifies deep transfer learning (DTL) into different four categories: instances-based DTL, mapping-based DTL, network-based DTL, and adversarial-based DTL. Some DNN-based systems have successfully applied deep transfer learning, including TTS [31], image classification [34] [35], machine translation [36] [37], automatic speech recognition [38] [39] [40], language identification [41], and sentiment classification [42].…”

LRSpeech

“…We randomly select 200 sentences for IR test and 20 sentences for MOS test, following the same test configuration in English. Each audio is listened by at least 5 testers for IR test and 9 The audio samples and complete experiments results on IR and MOS for TTS, and WER and CER for ASR can be founded in https://speechresearch.github.io/lrspeech. 20 testers for MOS test, who are all native Lithuanian speakers.…”