FluentTTS: Text-dependent Fine-grained Style Control for Multi-style TTS

Kim, Chang-Hwan; Um, Seyun; Yoon, Hyungchan; Kang, Hong-Goo

doi:10.21437/interspeech.2022-988

Cited by 3 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Diffusion Models and Backdoors. Diffusion Models have attracted a lot of researchers, to propose different models (Ho, Jain, and Abbeel 2020;Ermon 2019, 2020;Karras et al 2022;Rombach et al 2022) and different applications (Xu et al 2022;Jeong et 2021; Popov et al 2021;Kim, Kim, and Yoon 2022;Kong et al 2021;Mei and Patel 2022;Ho et al 2022;Ruiz et al 2023). A lot of methods are proposed to deal with the slow sampling process (Song, Meng, and Ermon 2021;Lu et al 2022a,b;Zhao et al 2023;Karras et al 2022;) Though they achieve a huge success, they are vulnerable to backdoor attacks (Chou, Chen, and Ho 2023a;Chen, Song, and Li 2023;Chou, Chen, and Ho 2023b).…”

Section: Related Workmentioning

confidence: 99%

“…Generative AIs become increasingly popular due to their applications in different synthesis or editing tasks (Couairon et al 2023;Meng et al 2022;Zhang et al 2022). Among the different types of generative AI models, Diffusion Models (DM) (Ho, Jain, and Abbeel 2020;Song and Ermon 2019;Karras et al 2022) are the recent driving force because of their superior ability to produce high-quality and diverse samples in many domains (Xu et al 2022;Jeong et al 2021;Popov et al 2021;Kim, Kim, and Yoon 2022;Kong et al 2021;Mei and Patel 2022;Ho et al 2022), and their more stable training than the adversarial training in traditional Generative Adversarial Networks (Goodfellow et al 2014;Arjovsky, Chintala, and Bottou 2017;Miyato et al 2018).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shift

An,

Chou,

Zhang

et al. 2024

AAAI

View full text Add to dashboard Cite

Diffusion models (DM) have become state-of-the-art generative models because of their capability of generating high-quality images from noises without adversarial training. However, they are vulnerable to backdoor attacks as reported by recent studies. When a data input (e.g., some Gaussian noise) is stamped with a trigger (e.g., a white patch), the backdoored model always generates the target image (e.g., an improper photo). However, effective defense strategies to mitigate backdoors from DMs are underexplored. To bridge this gap, we propose the first backdoor detection and removal framework for DMs. We evaluate our framework Elijah on over hundreds of DMs of 3 types including DDPM, NCSN and LDM, with 13 samplers against 3 existing backdoor attacks. Extensive experiments show that our approach can have close to 100% detection accuracy and reduce the backdoor effects to close to zero without significantly sacrificing the model utility.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shift

An,

Chou,

Zhang

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…GAN-based models [2,19] have brought innovative contributions to TTS in the last decade using adversarial training strategy. Recently, another successful generative approach, diffusion-based methods [11,12,13,9], have been proposed in speech synthesis, as the diffusion methods have proved their effectiveness in various generation tasks [21,8,7]. Compared to GAN-based models, diffusion methods have advantages in impressive results as well as distribution coverage, a fixed training objective, and scalability.…”

Section: Related Workmentioning

confidence: 99%

“…However, to learn various speakers' characteristics for the multispeaker TTS, the TTS model requires sufficient length of recorded speech for each person. Previous works [9,14,15] trained their models using audiobook dataset read by several speakers with enough lengths of utterances, where it is difficult to generalise models for unseen speakers. To solve this problem, we suggest an effective strategy, a speaker feature binding loss, maintaining speaker characteristics of target voices in synthesised speech.…”

Section: Speaker Conditioning With Cross-modal Biometricsmentioning

confidence: 99%

See 1 more Smart Citation

The policy analysis profession in Korea

Lee¹

2023

Policy Analysis in South Korea

View full text Add to dashboard Cite

The goal of this work is zero-shot text-to-speech synthesis, with speaking styles and voices learnt from facial characteristics. Inspired by the natural fact that people can imagine the voice of someone when they look at his or her face, we introduce a face-styled diffusion text-to-speech (TTS) model within a unified framework learnt from visible attributes, called FACE-TTS. This is the first time that face images are used as a condition to train a TTS model.We jointly train cross-model biometrics and TTS models to preserve speaker identity between face images and generated speech segments. We also propose a speaker feature binding loss to enforce the similarity of the generated and the ground truth speech segments in speaker embedding space. Since the biometric information is extracted directly from the face image, our method does not require extra fine-tuning steps to generate speech from unseen and unheard speakers. We train and evaluate the model on the LRS3 dataset, an in-the-wild audio-visual corpus containing background noise and diverse speaking styles. The project page is https: //facetts.github.io.

show abstract

Deep learning-based expressive speech synthesis: a systematic review of approaches, challenges, and resources

Barakat,

Turk,

Demiroglu

2024

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

Speech synthesis has made significant strides thanks to the transition from machine learning to deep learning models. Contemporary text-to-speech (TTS) models possess the capability to generate speech of exceptionally high quality, closely mimicking human speech. Nevertheless, given the wide array of applications now employing TTS models, mere high-quality speech generation is no longer sufficient. Present-day TTS models must also excel at producing expressive speech that can convey various speaking styles and emotions, akin to human speech. Consequently, researchers have concentrated their efforts on developing more efficient models for expressive speech synthesis in recent years. This paper presents a systematic review of the literature on expressive speech synthesis models published within the last 5 years, with a particular emphasis on approaches based on deep learning. We offer a comprehensive classification scheme for these models and provide concise descriptions of models falling into each category. Additionally, we summarize the principal challenges encountered in this research domain and outline the strategies employed to tackle these challenges as documented in the literature. In the Section 8, we pinpoint some research gaps in this field that necessitate further exploration. Our objective with this work is to give an all-encompassing overview of this hot research area to offer guidance to interested researchers and future endeavors in this field.

show abstract

FluentTTS: Text-dependent Fine-grained Style Control for Multi-style TTS

Cited by 3 publications

References 0 publications

Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shift

Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shift

The policy analysis profession in Korea

Deep learning-based expressive speech synthesis: a systematic review of approaches, challenges, and resources

Contact Info

Product

Resources

About