Disentangled Speaker and Language Representations Using Mutual Information Minimization and Domain Adaptation for Cross-Lingual TTS

Abstract

We propose a method for obtaining disentangled speaker and language representations via mutual information minimization and domain adaptation for cross-lingual text-to-speech (TTS) synthesis. The proposed method extracts speaker and language embeddings from acoustic features by a speaker encoder and a language encoder. Then the proposed method applies domain adaptation on the two embeddings to obtain language-invariant speaker embedding and speaker-invariant language embedding. To get more disentangled representations, the proposed method further uses mutual information minimization between the two embeddings to remove entangled information within each embedding. Disentangled representations of speaker and language are critical for cross-lingual TTS synthesis since entangled representations make it difficult to maintain speaker identity information when changing the language representation and consequently causes performance degradation. We evaluate the proposed method using English and Japanese multi-speaker datasets with a total of 207 speakers. Experimental result demonstrates that the proposed method significantly improves the naturalness and speaker similarity of both intra-lingual and cross-lingual TTS synthesis. Furthermore, we show that the proposed method has a good capability of maintaining the speaker identity between languages.

Improving Cross-lingual Speech Synthesis with Triplet Training Scheme

“…All the texts are converted into IPA symbols via espeak 1 . 80-dimensional mel-spectrograms are extracted by using Hanning window with frame shift 10 ms and frame length 42.7 ms. Then, Kaldi toolkit [18] is utilized to do forcedalignment.…”

“…Building a cross-lingual TTS system is a task that requires the system to synthesize speech of a language foreign to a target speaker [1]. It is straightforward to build a multi-lingual TTS system with a multi-lingual corpus using end-to-end models [2,3,4], yet such corpus of a same speaker is often difficult to collect.…”

“…[5] uses adversarial training to remove speaker identity from phonetic representations. [1] employs mutual information minimization and domain adaptation to disentangle the learned language and speaker embedding, which improves naturalness and speaker similarity of the synthesized cross-lingual speech. Beside explicit disentanglement, others use unified phonetic representations to share pronunciation across languages, which helps to implicitly disentangle language and speaker.…”

Speaker-Independent Emotional Voice Conversion via Disentangled Representations

“…Here, reconstruction loss is used to ensure that the autoencoder architecture does not lose too much information. Recent studies [22], [23], [24], [25], [26], [27] have demonstrated that mutual information minimization is an effective method for extracting disentangled representations in various style transfer tasks. To achieve better disentanglement of the emotion representations and emotion-independent representations of the input speech, we incorporate mutual information minimization into the autoencoder training process.…”

Content-Context Factorized Representations for Automated Speech Recognition

“…However, such methods have largely remained absent from work in the speech domain, particularly in the field of automated speech recognition. Recently, Xin et al [13] and Gong et al [14] demonstrated the benefits of disentangled representations when generating speech from text, Wang et al [15] showed the same for voice conversation, while Kwon et al [4] demonstrated gains in speaker recognition. Thus, it is clear that explicitly disentangled representations have the potential to improve generalization in a wide range of speech-related problems.…”