Unsupervised Style and Content Separation by Minimizing Mutual Information for Speech Synthesis

Hu, Ting-Yao; Shrivastava, Ashish; Tuzel, Oncel; Dhir, Chandra

doi:10.1109/icassp40776.2020.9054591

Cited by 27 publications

(21 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mutual information (MI) measures the dependence of two random variables from the perspective of information [15,20]. Given two random variables X and Y , the MI I(X; Y ) between them is equivalent to the Kullback-Leibler (KL) divergence between their joint distribution, PX,Y , and the product of marginals, PX PY .…”

Section: Algorithm 1 Pseudocode For MI Estimator Trainingmentioning

confidence: 99%

“…where M can be any function that forces the two expectations in the above equation to be finite. The authors in [15] proposed the use of a deep neural network for M , which enables the MI between X and Y to be estimated by maximizing the lower bound in Eq. 3 with respect to M using gradient descent.…”

Section: Algorithm 1 Pseudocode For MI Estimator Trainingmentioning

confidence: 99%

“…Ideally, the style embedding should not be able to reconstruct the content vector (i.e., there should be no information about the content in the style embedding). To this end, Hu et al [15] minimized the mutual information (MI) between the style and content vectors. In addition, MI has been applied to other speech synthesis tasks, such as learning disentangled representations of the speakers and the languages they speak [16], and maximizing the mutual information between the text and acoustic features to strengthen the dependency between them [17].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Using multiple reference audios and style embedding constraints for speech synthesis

Chen¹,

Wang²,

Ling³

et al. 2021

Preprint

View full text Add to dashboard Cite

The end-to-end speech synthesis model can directly take an utterance as reference audio, and generate speech from the text with prosody and speaker characteristics similar to the reference audio. However, an appropriate acoustic embedding must be manually selected during inference. Due to the fact that only the matched text and speech are used in the training process, using unmatched text and speech for inference would cause the model to synthesize speech with low content quality. In this study, we propose to mitigate these two problems by using multiple reference audios and style embedding constraints rather than using only the target audio. Multiple reference audios are automatically selected using the sentence similarity determined by Bidirectional Encoder Representations from Transformers (BERT). In addition, we use "target" style embedding from a pre-trained encoder as a constraint by considering the mutual information between the predicted and "target" style embedding. The experimental results show that the proposed model can improve the speech naturalness and content quality with multiple reference audios and can also outperform the baseline model in ABX preference tests of style similarity.

show abstract

Section: Algorithm 1 Pseudocode For MI Estimator Trainingmentioning

confidence: 99%

Section: Algorithm 1 Pseudocode For MI Estimator Trainingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Using multiple reference audios and style embedding constraints for speech synthesis

Chen¹,

Wang²,

Ling³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Similar to GST-Tacotron, VAE-based methods also use a large dataset to train a style encoder. Recently, a mutual information based method is proposed in [21]. This method disentangle content and style information by minimizing mutual information between content vector and style vector using a mutual information neural estimator (MINE) [22].…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Learning for Multi-Style Speech Synthesis with Limited Data

Liang

Cao

Chen

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Existing multi-style speech synthesis methods require either style labels or large amounts of unlabeled training data, making data acquisition difficult. In this paper, we present an unsupervised multi-style speech synthesis method that can be trained with limited data. We leverage instance discriminator to guide a style encoder to learn meaningful style representations from a multi-style dataset. Furthermore, we employ information bottleneck to filter out style-irrelevant information in the representations, which can improve speech quality and style similarity. Our method is able to produce desirable speech using a fairly small dataset, where the baseline GST-Tacotron fails. ABX tests show that our model significantly outperforms GST-Tacotron in both emotional speech synthesis task and multi-speaker speech synthesis task. In addition, we demonstrate that our method is able to learn meaningful style features with only 50 training samples per style.

show abstract

“…In the case of speech processing, an ideal disentangled representation would be able to separate fine-grained factors such as speaker identity, noise, recording channels, and prosody [22], as well as the linguistic content. Thus, disentanglement will allow learning of salient and robust representations from the speech that are essential for applications including speech recognition [64], prosody transfer [77,87], speaker verification [66], speech synthesis [31,77], and voice conversion [32], among other applications.…”

Section: Learning Disentangled Representationmentioning

confidence: 99%

Privacy-preserving Voice Analysis via Disentangled Representations

Aloufi

Haddadi

Boyle

2020

Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop

View full text Add to dashboard Cite

Voice User Interfaces (VUIs) are increasingly popular and built into smartphones, home assistants, and Internet of Things (IoT) devices. Despite offering an always-on convenient user experience, VUIs raise new security and privacy concerns for their users. In this paper, we focus on attribute inference attacks in the speech domain, demonstrating the potential for an attacker to accurately infer a target user's sensitive and private attributes (e.g. their emotion, sex, or health status) from deep acoustic models. To defend against this class of attacks, we design, implement, and evaluate a user-configurable, privacy-aware framework for optimizing speechrelated data sharing mechanisms. Our objective is to enable primary tasks such as speech recognition and user identification, while removing sensitive attributes in the raw speech data before sharing it with a cloud service provider. We leverage disentangled representation learning to explicitly learn independent factors in the raw data. Based on a user's preferences, a supervision signal informs the filtering out of invariant factors while retaining the factors reflected in the selected preference. Our experimental evaluation over five datasets shows that the proposed framework can effectively defend against attribute inference attacks by reducing their success rates to approximately that of guessing at random, while maintaining accuracy in excess of 99% for the tasks of interest. We conclude that negotiable privacy settings enabled by disentangled representations can bring new opportunities for privacy-preserving applications.

show abstract

Unsupervised Style and Content Separation by Minimizing Mutual Information for Speech Synthesis

Cited by 27 publications

References 15 publications

Using multiple reference audios and style embedding constraints for speech synthesis

Using multiple reference audios and style embedding constraints for speech synthesis

Unsupervised Learning for Multi-Style Speech Synthesis with Limited Data

Privacy-preserving Voice Analysis via Disentangled Representations

Contact Info

Product

Resources

About