Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learning

Hwang, Dongseong; Misra, Ananya; Huo, Zhouyuan; Siddhartha, Nikhil; Garg, Shefali; Qiu, David; Sim, Khe Chai; Strohman, Trevor; Beaufays, Françoise; He, You

doi:10.48550/arxiv.2110.00165

Cited by 6 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[24] also designs a similar hybrid multitask learning to train acoustic models under low-resource settings, comprising of supervised CTC, attention and self-supervised reconstruction losses. Similarly, [25] combines Fig. 1: An overview of our JUST framework.…”

Section: Related Workmentioning

confidence: 99%

Joint Unsupervised and Supervised Training for Multilingual ASR

Bai¹,

Li²,

Bapna³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Self-supervised training has shown promising gains in pretraining models and facilitating the downstream finetuning for speech recognition, like multilingual ASR. Most existing methods adopt a 2-stage scheme where the self-supervised loss is optimized in the first pretraining stage, and the standard supervised finetuning resumes in the second stage. In this paper, we propose an end-to-end (E2E) Joint Unsupervised and Supervised Training (JUST) method to combine the supervised RNN-T loss and the self-supervised contrastive and masked language modeling (MLM) losses. We validate its performance on the public dataset Multilingual LibriSpeech (MLS), which includes 8 languages and is extremely imbalanced. On MLS, we explore (1) JUST trained from scratch, and (2) JUST finetuned from a pretrained checkpoint. Experiments show that JUST can consistently outperform other existing state-of-the-art methods, and beat the monolingual baseline by a significant margin, demonstrating JUST's capability of handling low-resource languages in multilingual ASR. Our average WER of all languages outperforms average monolingual baseline by 33.3%, and the state-of-the-art 2-stage XLSR by 32%. On low-resource languages like Polish, our WER is less than half of the monolingual baseline and even beats the supervised transfer learning method which uses external supervision.

show abstract

Section: Related Workmentioning

confidence: 99%

Joint Unsupervised and Supervised Training for Multilingual ASR

Bai¹,

Li²,

Bapna³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Domain Adaptation (DA) for building efficient ASR systems has been a well-studied topic in literature with early work focusing on regularization [24,25], teacher-student learning [26,27] or adversarial learning [28,29]. Lately, unsupervised domain adaptation of ASR models has been gaining traction and researchers have been trying to find ways to use huge amounts of unlabeled data from the target domain for DA [27,30,31]. Continued pre-training is another common approach used [32].…”

Section: Domain Adaptationmentioning

confidence: 99%

PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations

Prasad¹,

Ghosh²,

Umesh³

2022

Preprint

View full text Add to dashboard Cite

While self-supervised speech representation learning (SSL) models serve a variety of downstream tasks, these models have been observed to overfit to the domain from which the unlabelled data originates. To alleviate this issue, we propose PADA (Pruning Assisted Domain Adaptation), and zero out redundant weights from models pre-trained on large amounts of outof-domain (OOD) data. Intuitively, this helps to make space for the target-domain ASR finetuning. The redundant weights can be identified through various pruning strategies which have been discussed in detail as a part of this work. Specifically, we investigate the effect of the recently discovered Task-Agnostic and Task-Aware pruning on PADA and propose a new pruning paradigm based on the latter, which we call Cross-Domain Task-Aware Pruning (CD-TAW). CD-TAW obtains the initial pruning mask from a well fine-tuned OOD model, which makes it starkly different from the rest of the pruning strategies discussed in the paper. Our proposed CD-TAW methodology achieves up to 20.6% relative WER improvement over our baseline when fine-tuned on a 2-hour subset of Switchboard data without language model (LM) decoding. Furthermore, we conduct detailed analysis to highlight the key design choices of our proposed method.

show abstract

“…Filtering noisy labels has a long history and can be seen as a form of curriculum learning [23]. More recently it has been explored in semi-supervised settings in ASR [24]. See [25,26] for more detailed surveys.…”

Section: Introductionmentioning

confidence: 99%

Production federated keyword spotting via distillation, filtering, and joint federated-centralized training

Hard¹,

Partridge²,

Chen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We trained a keyword spotting model using federated learning on real user devices and observed significant improvements when the model was deployed for inference on phones. To compensate for data domains that are missing from on-device training caches, we employed joint federated-centralized training. And to learn in the absence of curated labels on-device, we formulated a confidence filtering strategy based on user-feedback signals for federated distillation. These techniques created models that significantly improved quality metrics in offline evaluations and user-experience metrics in live A/B experiments.

show abstract

Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learning

Cited by 6 publications

References 0 publications

Joint Unsupervised and Supervised Training for Multilingual ASR

Joint Unsupervised and Supervised Training for Multilingual ASR

PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations

Production federated keyword spotting via distillation, filtering, and joint federated-centralized training

Contact Info

Product

Resources

About