Variational Attention Using Articulatory Priors for Generating Code Mixed Speech Using Monolingual Corpora

Abstract

Code Mixing-phenomenon where lexical items from one language are embedded in the utterance of another-is relatively frequent in multilingual communities and therefore speech systems should be able to process such content. However, building a voice capable of synthesizing such content typically requires bilingual recordings from the speaker which might not always be easy to obtain. In this work, we present an approach for building mixed lingual systems using only monolingual corpora. Specifically we present a way to train multi speaker text to speech system by incorporating stochastic latent variables into the attention mechanism with the objective of synthesizing code mixed content. We subject the prior distribution for such latent variables to match articulatory constraints. Subjective evaluation shows that our systems are capable of generating high quality synthesis in code mixed scenarios.

Importance of Supra-Segmental Information and Self-Supervised Framework for Spoken Language Diarization Task

“…The Gaussian posterior (GP) and i-vector approaches are the implicit approaches, whereas the phoneme posterior sequence (PS) extracted from the n-gram model is an explicit approach. In [16] and [17], the work uses bottleneck features (BNF) extracted from the trained ASR as the language representation and latent features with variational Bayes encoder to perform CSD, CSUD, and LD tasks. In [10], [18], [19] and [13], the works use deep learning architectures like the transformer, deepspeech2, and x-vector with deep clustering to implicitly model the language information for performing SLID and CSUD tasks.…”

Latent linguistic embedding for cross-lingual text-to-speech and voice conversion

“…Unsupervised speaker adaptation methods, such as speaker-adaptive TTS model conditioning on neural speaker embedding [17], have also shown promising results for the cross-lingual scenario. Cross-lingual TTS is also the foundation for more interesting applications such as code-mixing speech synthesis [18,19].…”