Non-Autoregressive TTS with Explicit Duration Modelling for Low-Resource Highly Expressive Speech

Shah, Raahil; Pokora, Kamil; Ezzerg, Abdelhamid; Klimkov, Viacheslav; Huybrechts, Goeric; Putrycz, Bartosz; Korzekwa, Daniel; Merritt, Thomas

doi:10.21437/ssw.2021-17

Cited by 14 publications

(14 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Without the Universal Vocoder, it would not be possible to generate the raw audio signal for hundreds of speakers included in the LibriTTS corpus. Details of the S2S method are shown in the works of Shah et al [20] and Jiao et al [66]. The main difference between these two models and our S2S model is the use of the P2P mapping to introduce pronunciation errors.…”

Section: Methodsmentioning

confidence: 99%

“…Huang et al [62] use a machine translation technique to generate text to train an ASR language model in a low-resource language. At the same time, Shah et al [20] and Huybrechts et al [19] employ S2S voice conversion to improve the quality of speech synthesis in the data reduction scenario.…”

Section: Synthetic Speech Generation For Pronunciation Error Detectionmentioning

confidence: 99%

“…The probability of pronunciation errors for all the words in a sentence can then be calculated using the Bayes rule [18]. In this new formulation, we move the complexity to learning the speech generation process that is well suited to the problem of limited speech availability [19][20][21]. The proposed method outperforms the state-of-the-art model [9] in detecting pronunciation errors in AUC metric by 41% from 0.528 to 0.749 on the GUT Isle Corpus of L2 Polish speakers.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Computer-assisted Pronunciation Training -- Speech synthesis is almost all you need

Korzekwa,

Lorenzo-Trueba,

Drugman

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

The research community has long studied computer-assisted pronunciation training (CAPT) methods in non-native speech. Researchers focused on studying various model architectures, such as Bayesian networks and deep learning methods, as well as on the analysis of different representations of the speech signal. Despite significant progress in recent years, existing CAPT methods are not able to detect pronunciation errors with high accuracy (only 60% precision at 40%-80% recall). One of the key problems is the low availability of mispronounced speech that is needed for the reliable training of pronunciation error detection models. If we had a generative model that could mimic non-native speech and produce any amount of training data, then the task of detecting pronunciation errors would be much easier. We present three innovative techniques based on phoneme-to-phoneme (P2P), text-to-speech (T2S), and speech-to-speech (S2S) conversion to generate correctly pronounced and mispronounced synthetic speech. We show that these techniques not only improve the accuracy of three machine learning models for detecting pronunciation errors but also help establish a new state-of-the-art in the field. Earlier studies have used simple speech generation techniques such as P2P conversion, but only as an additional mechanism to improve the accuracy of pronunciation error detection. We, on the other hand, consider speech generation to be the first-class method of detecting pronunciation errors. The effectiveness of these techniques is assessed in the tasks of detecting pronunciation and lexical stress errors. Non-native English speech corpora of German, Italian, and Polish speakers are used in the evaluations. The best proposed S2S technique improves the accuracy of detecting pronunciation errors in AUC metric by 41% from 0.528 to 0.749 compared to the state-of-the-art approach.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Synthetic Speech Generation For Pronunciation Error Detectionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Computer-assisted Pronunciation Training -- Speech synthesis is almost all you need

Korzekwa,

Lorenzo-Trueba,

Drugman

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This paradigm greatly enhanced naturalness and flexibility of speech synthesis. It enables new applications such as expressive [2,3] and low-resource [4] speech generation, speaker identity [5] and prosody transplantation [6,7]. This paper focuses on expressive speech synthesis, i.e.…”

Section: Introductionmentioning

confidence: 99%

On Granularity of Prosodic Representations in Expressive Text-to-Speech

Babianski

Pokora

Shah

et al. 2023

2022 IEEE Spoken Language Technology Workshop (SLT)

Self Cite

View full text Add to dashboard Cite

In expressive speech synthesis it is widely adopted to use latent prosody representations to deal with variability of the data during training. Same text may correspond to various acoustic realizations, which is known as a one-to-many mapping problem in text-to-speech. Utterance, word, or phoneme-level representations are extracted from target signal in an auto-encoding setup, to complement phonetic input and simplify that mapping. This paper compares prosodic embeddings at different levels of granularity and examines their prediction from text. We show that utterance-level embeddings have insufficient capacity and phoneme-level tend to introduce instabilities when predicted from text. Wordlevel representations impose balance between capacity and predictability. As a result, we close the gap in naturalness by 90% between synthetic speech and recordings on LibriTTS dataset, without sacrificing intelligibility.

show abstract

“…On the contrary to the frame-by-frame prediction of Mel spectrogram, non-autoregressive models generate Mel spectrogram parallelly, thus avoiding the error propagation through previously predicted frames. There has been limited research conducted on expressivity in non-autoregressive TTS [11,12]. Moreover in [13], authors proposed to leverage the normalizing Flow approach conditioned by the speaker identity.…”

Section: Introductionmentioning

confidence: 99%

Analysis of expressivity transfer in non-autoregressive end-to-end multispeaker TTS systems

Kulkarni¹,

Colotte²,

Jouvet³

2022

Interspeech 2022

View full text Add to dashboard Cite

The main objective of this work is to study the expressivity transfer in a speaker's voice for which no expressive speech data is available in non-autoregressive end-to-end TTS systems. We investigated the expressivity transfer capability of probability density estimation based on deep generative models, namely Generative Flow (Glow) and diffusion probabilistic models (DPM). The usage of deep generative models provides better log likelihood estimates and tractability of the system, subsequently providing high-quality speech synthesis with faster inference speed. Furthermore, we propose the usage of various expressivity encoders, which assist in expressivity transfer in the text-to-speech (TTS) system. More precisely, we used self-attention statistical pooling and multi-scale expressivity encoder architectures for creating a meaningful representation of expressivity.In addition to traditional subjective metrics used for speech synthesis evaluation, we incorporated cosine-similarity to measure the strength of attributes associated with speaker and expressivity. The performance of a non-autoregressive TTS system with a multi-scale expressivity encoder showed better expressivity transfer on Glow and DPM-based decoders. Thus, illustrating the ability of multi-scale architecture to apprehend the underlying attributes of expressivity from multiple acoustic features.

show abstract

Non-Autoregressive TTS with Explicit Duration Modelling for Low-Resource Highly Expressive Speech

Cited by 14 publications

References 31 publications

Computer-assisted Pronunciation Training -- Speech synthesis is almost all you need

Computer-assisted Pronunciation Training -- Speech synthesis is almost all you need

On Granularity of Prosodic Representations in Expressive Text-to-Speech

Analysis of expressivity transfer in non-autoregressive end-to-end multispeaker TTS systems

Contact Info

Product

Resources

About