Federated Acoustic Modeling for Automatic Speech Recognition

Cui, Xiaodong; Kingsbury, Brian

doi:10.1109/icassp39728.2021.9414305

Cited by 34 publications

(16 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…FL-based adaptation for ASR models faces several unique challenges including the lack of ground truth transcriptions, high compute and cross-device network communication costs, the non independent and identical distribution of data (non-IIDness), and the difficulty of providing privacy guarantees. Several recent works have considered cross-device FL for ASR applications [14,15,16,17,18,19]. In particular, the challenge of training on non-IID data has been addressed using weighted model averaging [14,15] and federated variation noise [17].…”

Section: Introductionmentioning

confidence: 99%

“…1 This paper was submitted to Interspeech 2022. A common assumption made by many existing works on cross-device FL for ASR is the availability of ground-truth transcriptions [14,15,16,17,18,19]. In reality, however, users of on-device ASR applications have neither the incentive to transcribe their audios, nor the desire to frequently edit inaccurate automatic transcriptions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Federated Domain Adaptation for ASR with Full Self-Supervision

Jia¹,

Mahadeokar²,

Zheng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Cross-device federated learning (FL) protects user privacy by collaboratively training a model on user devices, therefore eliminating the need for collecting, storing, and manually labeling user data. Previous works have considered cross-device FL for automatic speech recognition (ASR), however, there are a few important challenges that have not been fully addressed. These include the lack of ground-truth ASR transcriptions, and the scarcity of compute resource and network bandwidth on edge devices. In this paper, we address these two challenges. First, we propose a federated learning system to support ondevice ASR adaptation with full self-supervision, which uses self-labeling together with data augmentation and filtering techniques. The proposed system can improve a strong Emformer-Transducer based ASR model pretrained on out-of-domain data, using in-domain audios without any ground-truth transcriptions. Second, to reduce the training cost, we propose a self-restricted RNN Transducer (SR-RNN-T) loss, a new variant of alignmentrestricted RNN-T that uses Viterbi forced-alignment from selfsupervision. To further reduce the compute and network cost, we systematically explore adapting only a subset of weights in the Emformer-Transducer. Our best training recipe achieves a 12.9% relative WER reduction over the strong out-of-domain baseline, which equals 70% of the reduction achievable with full human supervision and centralized training.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Federated Domain Adaptation for ASR with Full Self-Supervision

Jia¹,

Mahadeokar²,

Zheng³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Various Federated ASR methods have been proposed to train ASR models in FL systems [11,12,13,14]. Specific challenges arising from data heterogeneity (speech characteristics, amount of data, acoustic environments etc.)…”

Section: Introductionmentioning

confidence: 99%

“…Specific challenges arising from data heterogeneity (speech characteristics, amount of data, acoustic environments etc.) are addressed via client-dependent data transformations [14] and imposing upper limits on the number of client samples [13]. Improvements to distributed optimization of models, such as Word Error Rate (WER) based aggregation [12] and a hierarchical gradient weighting scheme [11] have been proposed.…”

Section: Introductionmentioning

confidence: 99%

FedNST: Federated Noisy Student Training for Automatic Speech Recognition

Mehmood¹,

Dobrowolska²,

Saravanan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Federated Learning (FL) enables training state-of-the-art Automatic Speech Recognition (ASR) models on user devices (clients) in distributed systems, hence preventing transmission of raw user data to a central server. A key challenge facing practical adoption of FL for ASR is obtaining ground-truth labels on the clients. Existing approaches rely on clients to manually transcribe their speech, which is impractical for obtaining large training corpora. A promising alternative is using semi-/selfsupervised learning approaches to leverage unlabelled user data. To this end, we propose a new Federated ASR method called FEDNST for noisy student training of distributed ASR models with private unlabelled user data. We explore various facets of FEDNST, such as training models with different proportions of unlabelled and labelled data, and evaluate the proposed approach on 1173 simulated clients. Evaluating FEDNST on Lib-riSpeech, where 960 hours of speech data is split equally into server (labelled) and client (unlabelled) data, showed a 22.5% relative word error rate reduction (WERR) over a supervised baseline trained only on server data.

show abstract

“…For example, in [14], federated learning is applied with an end-to-end (E2E) model on the French set of the Common Voice dataset [15]. In [16], an RNN-T architecture ASR [17] is used on the LibriSpeech corpus, and [18] applies FL to a hybrid ASR model. Additionally, in [19] Dimitriadis et.al.…”

Section: Introductionmentioning

confidence: 99%

Federated Learning in ASR: Not as Easy as You Think

Yu,

Freiwald,

Tewes

et al. 2021

Preprint

View full text Add to dashboard Cite

With the growing availability of smart devices and cloud services, personal speech assistance systems are increasingly used on a daily basis. Most devices redirect the voice recordings to a central server, which uses them for upgrading the recognizer model. This leads to major privacy concerns, since private data could be misused by the server or third parties. Federated learning is a decentralized optimization strategy that has been proposed to address such concerns. Utilizing this approach, private data is used for on-device training. Afterwards, updated model parameters are sent to the server to improve the global model, which is redistributed to the clients. In this work, we implement federated learning for speech recognition in a hybrid and an end-toend model. We discuss the outcomes of these systems, which both show great similarities and only small improvements, pointing to a need for a deeper understanding of federated learning for speech recognition.

show abstract

Federated Acoustic Modeling for Automatic Speech Recognition

Cited by 34 publications

References 13 publications

Federated Domain Adaptation for ASR with Full Self-Supervision

Federated Domain Adaptation for ASR with Full Self-Supervision

FedNST: Federated Noisy Student Training for Automatic Speech Recognition

Federated Learning in ASR: Not as Easy as You Think

Contact Info

Product

Resources

About