Large-Scale ASR Domain Adaptation Using Self- and Semi-Supervised Learning

Hwang, Dongseong; Misra, Ananya; Huo, Zhouyuan; Siddhartha, Nikhil; Garg, Shefali; Qiu, David; Sim, Khe Chai; Strohman, Trevor; Beaufays, Françoise; He, Yanzhang

doi:10.1109/icassp43922.2022.9746719

Cited by 22 publications

(8 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Method Model Adaptation Setting Language [23], [25], [26] Teacher-Student Hard and soft labels Conformer RNN-T [27] Transformer CTC RNN-T [19] News speech, Voice search, Far-field, Telephony, YouTube English [4], [5] Teacher-Student Soft labels TDNN-LSTM [28] Noise, Far-field English [29] Teacher-Student Hard and soft labels NiN-CNN [30] Dialects Children speech Japanese [31] Teacher-Student Soft labels Streaming RNN-T [32] Multilingual English, Brazilian Portuguese, Russian, , Turkish, Nordic/Germanic [6], [33], [34] Domain Adversarial Training TDNN Kaldi [35], [36] DNN-HMM DNN-HMM Noise, Channel English [37] Domain Adversarial Training RNN-CTC [38] Far-field English [8], [39 1) Inspired by recent advances on UDA for Natural Language Processing systems [45], we propose a finetuning strategy for speech models, where the self-supervised objective is based on a contrastive loss in Section III. Contrary to prior works, who leverage only in-domain self-supervision, we find that in this contrastive setting this leads to mode-collapse of the latent representations, and mixed source and target domain self-supervision is essential.…”

Section: Workmentioning

confidence: 99%

“…The agreement between model predictions with and without dropout are used for confidence scoring. In [23] a multi-task training objective with a confidence loss is applied to minimise the binary cross entropy between the estimated confidence and the binary target sequence. In order to learn more robust and generalizable features from the teacher model, Noisy Student Training (NST) has been proposed in [52].…”

Section: B Teacher-student Modelsmentioning

confidence: 99%

“…Modern ASR pipelines, increasingly rely on end-to-end neural networks, e.g., [18], [19], or large pretrained models with self-supervised objectives [20], [21]. The key approaches employed for UDA of end-to-end ASR models can be grouped in three categories, namely, teacher-student learning [10], domain adversarial training [22], and target domain selfsupervision [23]. The benefit of these techniques is that they do not require any special knowledge about the source or the target domain.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems: A case study for Modern Greek

Paraskevopoulos¹,

Kouzelis²,

Rouvalis³

et al. 2023

Preprint

View full text Add to dashboard Cite

<p>Modern speech recognition systems exhibits rapid performance degradation under domain shift. This issue is especially prevalent in data-scarce settings, such as low-resource languages, where diversity of training data is limited.</p> <p>In this work we propose M2DS2, a simple and sample-efficient finetuning strategy for large pretrained speech models, based on mixed source and target domain self-supervision. We find that including source domain self-supervision stabilizes training and avoids mode collapse of the latent representations. For evaluation, we collect HParl, a 120 hour speech corpus for Greek, consisting of plenary sessions in the Greek Parliament. We merge HParl with two popular Greek corpora to create GREC-MD, a test-bed for multi-domain evaluation of Greek ASR systems. In our experiments we find that, while other Unsupervised Domain Adaptation baselines fail in this resource-constrained environment, M2DS2 yields significant improvements for cross-domain adaptation, even when a only a few hours of in-domain audio are available. When we relax the problem in a weakly supervised setting, we find that independent adaptation for audio using M2DS2 and language using simple LM augmentation techniques is particularly effective, yielding word error rates comparable to the fully supervised baselines.</p>

show abstract

Section: Workmentioning

confidence: 99%

Section: B Teacher-student Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems: A case study for Modern Greek

Paraskevopoulos¹,

Kouzelis²,

Rouvalis³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…We assume that G is much larger than T . Confidence filtering is a classic method in data selection for ASR [28,29], and has been successfully applied to end-to-end models [12,30,31]. However, confidence methods focus on data of good transcript quality, which might not be useful for self-supervised pre-training.…”

Section: Data Selection For Self-supervised Learningmentioning

confidence: 99%

Unsupervised Data Selection via Discrete Speech Representation for ASR

Lü¹,

Wang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Self-supervised learning of speech representations has achieved impressive results in improving automatic speech recognition (ASR). In this paper, we show that data selection is important for self-supervised learning. We propose a simple and effective unsupervised data selection method which selects acoustically similar speech to a target domain. It takes the discrete speech representation available in common self-supervised learning frameworks as input, and applies a contrastive data selection method on the discrete tokens. Through extensive empirical studies we show that our proposed method reduces the amount of required pre-training data and improves the downstream ASR performance. Pre-training on a selected subset of 6% of the general data pool results in 11.8% relative improvements in LibriSpeech test-other compared to pre-training on the full set. On Multilingual LibriSpeech French, German, and Spanish test sets, selecting 6% data for pre-training reduces word error rate by more than 15% relatively compared to the full set, and achieves competitive results compared to current state-of-theart performances.

show abstract

“…A foundation model [1] is usually a big model trained on broad data (generally using self-supervision at scale) that can be fine-tuned to a wide range of downstream tasks and has aroused extensive attention due to its impressive quality improvements and emergent capabilities [2,3,4,5]. In speech community, self-supervised pretraining speech foundation models on a large amount of unsupervised speech has shown impressive quality improvements on various speech recognition tasks [6,7]. There are two main categories of speech self-supervised learning algorithms.…”

Section: Introductionmentioning

confidence: 99%

Resource-Efficient Transfer Learning From Speech Foundation Model Using Hierarchical Feature Fusion

Huo¹,

Sim²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Self-supervised pre-training of a speech foundation model, followed by supervised fine-tuning, has shown impressive quality improvements on automatic speech recognition (ASR) tasks. Fine-tuning separate foundation models for many downstream tasks are expensive since the foundation model is usually very big. Parameterefficient fine-tuning methods (e.g. adapter, sparse update methods) offer an alternative paradigm where a small set of parameters are updated to adapt the foundation model to new tasks. However, these methods still suffer from a high computational memory cost and slow training speed because they require backpropagation through the entire neural network at each step. In the paper, we analyze the performance of features at different layers of a foundation model on the speech recognition task and propose a novel hierarchical feature fusion method for resource-efficient transfer learning from speech foundation models. Experimental results show that the proposed method can achieve better performance on speech recognition task than existing algorithms with fewer number of trainable parameters, less computational memory cost and faster training speed. After combining with Adapters at all layers, the proposed method can achieve the same performance as fine-tuning the whole model with 97% fewer trainable encoder parameters and 53% faster training speed.

show abstract

Large-Scale ASR Domain Adaptation Using Self- and Semi-Supervised Learning

Cited by 22 publications

References 12 publications

Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems: A case study for Modern Greek

Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems: A case study for Modern Greek

Unsupervised Data Selection via Discrete Speech Representation for ASR

Resource-Efficient Transfer Learning From Speech Foundation Model Using Hierarchical Feature Fusion

Contact Info

Product

Resources

About