Code-Switched Speech Synthesis Using Bilingual Phonetic Posteriorgram with Only Monolingual Corpora

Cao, Yuewen; Liu, Songxiang; Wu, Xixin; Kang, Shiyin; Liu, Peng; Wu, Zhiyong; Liu, Xunying; Su, Dan; Yu, Dong; Meng, Helen

doi:10.1109/icassp40776.2020.9053094

Cited by 16 publications

(13 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Maiti et al [11] approximated the domain shift by computing the distance between speaker embeddings of source and target language of a bilingual speaker and use this distance to convert speaker embedding in source language space to target language space. Cao et al [14] used bilingual phonetic posteriorgrams as a language-independent feature to synthesize cross-lingual speech. It removed the language discrepancy in the feature level.…”

Section: Introductionmentioning

confidence: 99%

Disentangled Speaker and Language Representations Using Mutual Information Minimization and Domain Adaptation for Cross-Lingual TTS

Xin

Komatsu

Takamichi

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

We propose a method for obtaining disentangled speaker and language representations via mutual information minimization and domain adaptation for cross-lingual text-to-speech (TTS) synthesis. The proposed method extracts speaker and language embeddings from acoustic features by a speaker encoder and a language encoder. Then the proposed method applies domain adaptation on the two embeddings to obtain language-invariant speaker embedding and speaker-invariant language embedding. To get more disentangled representations, the proposed method further uses mutual information minimization between the two embeddings to remove entangled information within each embedding. Disentangled representations of speaker and language are critical for cross-lingual TTS synthesis since entangled representations make it difficult to maintain speaker identity information when changing the language representation and consequently causes performance degradation. We evaluate the proposed method using English and Japanese multi-speaker datasets with a total of 207 speakers. Experimental result demonstrates that the proposed method significantly improves the naturalness and speaker similarity of both intra-lingual and cross-lingual TTS synthesis. Furthermore, we show that the proposed method has a good capability of maintaining the speaker identity between languages.

show abstract

Section: Introductionmentioning

confidence: 99%

Disentangled Speaker and Language Representations Using Mutual Information Minimization and Domain Adaptation for Cross-Lingual TTS

Xin

Komatsu

Takamichi

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Sun et al [8] proposed to use a speaker-independent speech recognition model to extract phonetic posteriorgrams (PPGs) from source speech (uttered by the source speaker) , and then use the target speaker data to map PPGs to the target speaker's speech. Cao et al [9] used PPGs in cross-language speech synthesis tasks, and the synthesized speech has been improved in terms of speech intelligibility and audio fidelity. In [10], the parameter space of the original speaker's voice was designed to be decomposed into two subspaces, one for modeling spoken content and the other one for the speaker 's voice.…”

Section: Introductionmentioning

confidence: 99%

Cloning one's voice using very limited data in the wild

Dai¹,

Chen²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

With the increasing popularity of speech synthesis products, the industry has put forward more requirements for personalized speech synthesis: (1) How to use low-resource, easily accessible data to clone a person's voice. (2) How to clone a person's voice while controlling the style and prosody. To solve the above two problems, we proposed the Hieratron model framework in which the prosody and timbre are modeled separately using two modules, therefore, the independent control of timbre and the other characteristics of audio can be achieved while generating speech. The practice shows that, for very limited target speaker data in the wild, Hieratron has obvious advantages over the traditional method, in addition to controlling the style and language of the generated speech, the mean opinion score on speech quality of the generated speech has also been improved by more than 0.2 points.

show abstract

“…[10] proposes a phonetic transformation network to learn target symbol distribution with the help of Automatic Speech Recognition (ASR) systems. In [11,12], language-independent Phonetic PosteriorGram (PPG) features of ASR models are used as input for cross-lingual TTS models. [ further proposes a mixed-lingual grapheme-to-phoneme (G2P) frontend to improve the pronunciation of mixed-lingual sentences in cross-lingual TTS systems.…”

Section: Introductionmentioning

confidence: 99%

Exploring Timbre Disentanglement in Non-Autoregressive Cross-Lingual Text-to-Speech

Zhan¹,

Yu²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we present a FastPitch-based non-autoregressive cross-lingual Text-to-Speech (TTS) model built with languageindependent input representation and monolingual force aligners. We propose a phoneme length regulator that solves the length mismatch problem between language-independent phonemes and monolingual alignment results. Our experiments show that (1) an increasing number of training speakers encourages non-autoregressive cross-lingual TTS model to disentangle speaker and language representations, and (2) variance adaptors of FastPitch model can help disentangle speaker identity from learned representations in crosslingual TTS. The subjective evaluation shows that our proposed model is able to achieve decent speaker consistency and similarity. We further improve the naturalness of Mandarin-dominated mixedlingual utterances by utilizing the controllability of our proposed model.

show abstract

Code-Switched Speech Synthesis Using Bilingual Phonetic Posteriorgram with Only Monolingual Corpora

Cited by 16 publications

References 16 publications

Disentangled Speaker and Language Representations Using Mutual Information Minimization and Domain Adaptation for Cross-Lingual TTS

Disentangled Speaker and Language Representations Using Mutual Information Minimization and Domain Adaptation for Cross-Lingual TTS

Cloning one's voice using very limited data in the wild

Exploring Timbre Disentanglement in Non-Autoregressive Cross-Lingual Text-to-Speech

Contact Info

Product

Resources

About