Improving Cross-Lingual Speech Synthesis with Triplet Training Scheme

Ye, Jianhao; Zhou, Hongbin; Su, Zhiba; He, Wendi; Ren, Kaimeng; Li, Lin; Lu, Hanqing

doi:10.1109/icassp43922.2022.9746914

Cited by 5 publications

(9 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This task has many applications, such as code-mixed speech synthesis for a voice agent, foreign movie dubbing [4], and computer-assisted pronunciation teaching [5]. Due to the difficulty of obtaining a bilingual corpus produced by a highly proficient speaker in both languages, more practically, current studies mainly build a cross-lingual TTS system based on corpora from monolingual speakers in different languages [6], [7], [8], [9]. However, these approaches mostly ignore modeling emotion aspects during speech generation, while emotion is a kind of indispensable paralinguistic information that reveals the speaker's intentions and moods.…”

Section: Introductionmentioning

confidence: 99%

“…The reason for this phenomenon is that each speaker in the training set speaks only one language, and the entanglement between different speech factors, such as linguistic content, speaker identity, and emotion, makes it hard to only transfer the speaker's timbre across different languages. Therefore, the key to alleviating this foreign accent issue is how to properly disentangle the speaker and language or linguistic content [8], [9], [10].…”

Section: Introductionmentioning

confidence: 99%

“…Based on the disentanglement strategy, the existing crosslingual approaches can be roughly divided into implicitbased and explicit-based methods [9]. Implicit-based methods mainly study the unified linguistic/phonetic representations across languages to disentangle language and speaker timbre implicitly [11], [12], [13], [14], [15], [16].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

DiCLET-TTS: Diffusion Model Based Cross-Lingual Emotion Transfer for Text-to-Speech — A Study Between English and Mandarin

Li,

Hu,

Cong

et al. 2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

While the performance of cross-lingual TTS based on monolingual corpora has been significantly improved recently, generating cross-lingual speech still suffers from the foreign accent problem, leading to limited naturalness. Besides, current cross-lingual methods ignore modeling emotion, which is indispensable paralinguistic information in speech delivery. In this paper, we propose DiCLET-TTS, a Diffusion model based Cross-Lingual Emotion Transfer method that can transfer emotion from a source speaker to the intra-and cross-lingual target speakers. Specifically, to relieve the foreign accent problem while improving the emotion expressiveness, the terminal distribution of the forward diffusion process is parameterized into a speaker-irrelevant but emotion-related linguistic prior by a prior text encoder with the emotion embedding as a condition. To address the weaker emotional expressiveness problem caused by speaker disentanglement in emotion embedding, a novel orthogonal projection based emotion disentangling module (OP-EDM) is proposed to learn the speaker-irrelevant but emotiondiscriminative embedding. Moreover, a condition-enhanced DPM decoder is introduced to strengthen the modeling ability of the speaker and the emotion in the reverse diffusion process to further improve emotion expressiveness in speech delivery. Cross-lingual emotion transfer experiments show the superiority of DiCLET-TTS over various competitive models and the good design of OP-EDM in learning speaker-irrelevant but emotiondiscriminative embedding.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

DiCLET-TTS: Diffusion Model Based Cross-Lingual Emotion Transfer for Text-to-Speech — A Study Between English and Mandarin

Li,

Hu,

Cong

et al. 2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

show abstract

Section: Introductionmentioning

confidence: 99%

“…Over the recent years, various approaches have been proposed to build cross-lingual Text-to-Speech (TTS) systems [1][2][3]. However, L2 (second-language) accents frequently appear in such cross-lingual scenarios, and several attempts have been made to improve its nativeness [1][2][3]. To make matters even worse, subjective evaluation of less major languages is challenging, especially for researchers in a less diverse environment.…”

Section: Introductionmentioning

confidence: 99%

An Empirical Study on L2 Accents of Cross-lingual Text-to-Speech Systems via Vowel Space

Lee¹,

Bae²,

Mun³

et al. 2022

Preprint

View full text Add to dashboard Cite

With the recent developments in cross-lingual Text-to-Speech (TTS) systems, L2 (second-language, or foreign) accent problems arise. Moreover, running a subjective evaluation for such cross-lingual TTS systems is troublesome. The vowel space analysis, which is often utilized to explore various aspects of language including L2 accents, is a great alternative analysis tool. In this study, we apply the vowel space analysis method to explore L2 accents of cross-lingual TTS systems. Through the vowel space analysis, we observe the three followings: a) a parallel architecture (Glow-TTS) is less L2-accented than an auto-regressive one (Tacotron); b) L2 accents are more dominant in non-shared vowels in a language pair; and c) L2 accents of cross-lingual TTS systems share some phenomena with those of human L2 learners. Our findings imply that it is necessary for TTS systems to handle each language pair differently, depending on their linguistic characteristics such as non-shared vowels. They also hint that we can further incorporate linguistics knowledge in developing cross-lingual TTS systems.

show abstract

Mels-Tts : Multi-Emotion Multi-Lingual Multi-Speaker Text-To-Speech System Via Disentangled Style Tokens

Choi,

Bae,

Lee

et al. 2024

ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Improving Cross-Lingual Speech Synthesis with Triplet Training Scheme

Cited by 5 publications

References 9 publications

DiCLET-TTS: Diffusion Model Based Cross-Lingual Emotion Transfer for Text-to-Speech — A Study Between English and Mandarin

DiCLET-TTS: Diffusion Model Based Cross-Lingual Emotion Transfer for Text-to-Speech — A Study Between English and Mandarin

An Empirical Study on L2 Accents of Cross-lingual Text-to-Speech Systems via Vowel Space

Mels-Tts : Multi-Emotion Multi-Lingual Multi-Speaker Text-To-Speech System Via Disentangled Style Tokens

Contact Info

Product

Resources

About