Conformer Parrotron: A Faster and Stronger End-to-End Speech Conversion and Recognition Model for Atypical Speech

Chen, Zhehuai; Ramabhadran, Bhuvana; Biadsy, Fadi; Zhang, Xia; Chen, Youzheng; Jiang, Liyang; Chu, Andrea; Doshi, Rohan; Mengibar, Pedro Jose Moreno

doi:10.21437/interspeech.2021-676

Cited by 7 publications

(7 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is consistent with our previous work that Parrotron model fine-tuning achieves high quality model personalization for atypical speech [4,23,18,12]. For this experiment, we employ the same model architecture and the same fine-tuning procedure described in [18]. This work confirms that a fine-tuning strategy performs well on speakers across etiologies and severities.…”

Section: Basemodel Vs Model Fine-tuning Resultssupporting

confidence: 87%

“…We find that fine-tuning the entire model (i.e., adapting all parameters of the Basemodel including encoder and decoders) for each speaker exerts substantial improvements across all our speakers with an average WER of 14.2%. This is consistent with our previous work that Parrotron model fine-tuning achieves high quality model personalization for atypical speech [4,23,18,12]. For this experiment, we employ the same model architecture and the same fine-tuning procedure described in [18].…”

Section: Basemodel Vs Model Fine-tuning Resultssupporting

confidence: 75%

“…It has been shown that fine-tuning the full model on hundreds of speech utterances from a single speaker obtains a one-to-one speech conversion model to convert from atypical to typical speech with significant reduction in Word Error Rate (WER) and MOS scores when compared to the Basemodel. [4,23,18].…”

Section: Parrotron Basemodelmentioning

confidence: 99%

“…Prior work has shown that fine-tuning the entire or a large subset of the parameter space can successfully personalize a speech model for end-to-end Automatic Speech Recognition (ASR) and speech conversion. [15,3,16,17,4,18,19,20]. To motivate our framework, we summarize the limitations of employing model fine-tuning for speech model personalization:…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Scalable Model Specialization Framework for Training and Inference using Submodels and its Application to Speech Model Personalization

Biadsy¹,

Youzheng²,

Zhang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Model fine-tuning and adaptation have become a common approach for model specialization for downstream tasks or domains. Fine-tuning the entire model or a subset of the parameters using light-weight adaptation has shown considerable success across different specialization tasks. Fine-tuning a model for a large number of domains typically requires starting a new training job for every domain posing scaling limitations. Once these models are trained, deploying them also poses significant scalability challenges for inference for real-time applications. In this paper, building upon prior light-weight adaptation techniques, we propose a modular framework that enables us to substantially improve scalability for model training and inference. We introduce Submodels that can be quickly and dynamically loaded for on-the-fly inference. We also propose multiple approaches for training those Submodels in parallel using an embedding space in the same training job. We test our framework on an extreme use-case which is speech model personalization for atypical speech, requiring a Submodel for each user. We obtain 128× Submodel throughput with a fixed computation budget without a loss of accuracy. We also show that learning a speaker-embedding space can scale further and reduce the amount of personalization training data required per speaker.

show abstract

Section: Basemodel Vs Model Fine-tuning Resultssupporting

confidence: 87%

Section: Basemodel Vs Model Fine-tuning Resultssupporting

confidence: 75%

Section: Parrotron Basemodelmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Scalable Model Specialization Framework for Training and Inference using Submodels and its Application to Speech Model Personalization

Biadsy¹,

Youzheng²,

Zhang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Rule-based VC tends to apply manually designed, speakerdependent rules to correct phoneme errors or modify temporal and frequency features to improve intelligibility [12,13]. Statistical VC automatically maps the features of dysarthric speech to those of normal speech, where typical approaches contain Gaussian mixture model [14], non-negative matrix factorization [15,16], partial least squares [17], and deep learning methods including sequenceto-sequence (seq2seq) models [18][19][20] and gated convolutional networks [21]. Though significant progress has been made, previous work generally ignores speaker identity preservation, which loses the ability for patients to demonstrate their personality via acoustic characteristics.…”

Section: Introductionmentioning

confidence: 99%

Speaker Identity Preservation in Dysarthric Speech Reconstruction by Adversarial Speaker Adaptation

Wang¹,

Liu²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Dysarthric speech reconstruction (DSR), which aims to improve the quality of dysarthric speech, remains a challenge, not only because we need to restore the speech to be normal, but also must preserve the speaker's identity. The speaker representation extracted by the speaker encoder (SE) optimized for speaker verification has been explored to control the speaker identity. However, the SE may not be able to fully capture the characteristics of dysarthric speakers that are previously unseen. To address this research problem, we propose a novel multi-task learning strategy, i.e., adversarial speaker adaptation (ASA). The primary task of ASA fine-tunes the SE with the speech of the target dysarthric speaker to effectively capture identityrelated information, and the secondary task applies adversarial training to avoid the incorporation of abnormal speaking patterns into the reconstructed speech, by regularizing the distribution of reconstructed speech to be close to that of reference speech with high quality. Experiments show that the proposed approach can achieve enhanced speaker similarity and comparable speech naturalness with a strong baseline approach. Compared with dysarthric speech, the reconstructed speech achieves 22.3% and 31.5% absolute word error rate reduction for speakers with moderate and moderate-severe dysarthria respectively. Our demo page is released here 1 .

show abstract

Speaker Identity Preservation in Dysarthric Speech Reconstruction by Adversarial Speaker Adaptation

Wang

Liu

et al. 2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Conformer Parrotron: A Faster and Stronger End-to-End Speech Conversion and Recognition Model for Atypical Speech

Cited by 7 publications

References 17 publications

A Scalable Model Specialization Framework for Training and Inference using Submodels and its Application to Speech Model Personalization

A Scalable Model Specialization Framework for Training and Inference using Submodels and its Application to Speech Model Personalization

Speaker Identity Preservation in Dysarthric Speech Reconstruction by Adversarial Speaker Adaptation

Speaker Identity Preservation in Dysarthric Speech Reconstruction by Adversarial Speaker Adaptation

Contact Info

Product

Resources

About