Training Keyword Spotting Models on Non-IID Data with Federated Learning

Hard, A. S.; Partridge, Kurt; Nguyen, Cameron; Subrahmanya, Niranjan; Shah, Aishanee; Zhu, Pai; Moreno, Ignacio López; Mathews, Rajiv

doi:10.48550/arxiv.2005.10406

Cited by 10 publications

(12 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Applications of Federated Learning in HAR. While many of the early use-cases of FL were related to natural language processing [21], visual recognition [24,49], and speech recognition [22] tasks, we are now also witnessing its applications in the area of human-activity recognition [3,18,48,62,63,77]. We extend this line of work on FL in HAR, albeit in the context of multi-device environments.…”

Section: Related Workmentioning

confidence: 89%

“…At a high level, FL involves repeating three steps: (i) updating the parameters of a shared prediction model locally on each remote client, (ii) sending the local parameter updates to a central server for aggregation, and (iii) receiving the aggregated prediction model back on the remote client for the next round of local updates. While many of the early use-cases of FL were related to natural language processing [21], visual recognition [24,49], and speech recognition [22] tasks, we are now also witnessing its applications in the area of human-activity recognition [17].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

FLAME: Federated Learning Across Multi-device Environments

Cho¹,

Mathur²,

Kawsar³

2022

Preprint

View full text Add to dashboard Cite

Federated Learning (FL) enables distributed training of machine learning models while keeping personal data on user devices private. While we witness increasing applications of FL in the area of mobile sensing, such as human-activity recognition, FL has not been studied in the context of a multi-device environment (MDE), wherein each user owns multiple data-producing devices. With the proliferation of mobile and wearable devices, MDEs are increasingly becoming popular in ubicomp settings, therefore necessitating the study of FL in them. FL in MDEs is characterized by high non-IID-ness across clients, complicated by the presence of both user and device heterogeneities. Further, ensuring efficient utilization of system resources on FL clients in a MDE remains an important challenge. In this paper, we propose FLAME, a user-centered FL training approach to counter statistical and system heterogeneity in MDEs, and bring consistency in inference performance across devices. FLAME features (i) user-centered FL training utilizing the time alignment across devices from the same user; (ii) accuracy-and efficiency-aware device selection; and (iii) model personalization to devices. We also present an FL evaluation testbed with realistic energy drain and network bandwidth profiles, and a novel class-based data partitioning scheme to extend existing HAR datasets to a federated setup. Our experiment results on three multi-device HAR datasets show that FLAME outperforms various baselines by 4.8-33.8% higher 𝐹 1 score, 1.02-2.86× greater energy efficiency, and up to 2.02× speedup in convergence to target accuracy through fair distribution of the FL workload.CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing systems and tools; • Computing methodologies → Cooperation and coordination.

show abstract

Section: Related Workmentioning

confidence: 89%

Section: Introductionmentioning

confidence: 99%

FLAME: Federated Learning Across Multi-device Environments

Cho¹,

Mathur²,

Kawsar³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, performing on-device federated training of acoustic models has attracted considerable attention [7,9,12,23,44]. In [23], FL was employed for a keyword spotting task and the development of a wake-word detection system, whereas, [7,9] investigated the efect of non-i.i.d. distributions on the same task.…”

Section: Related Workmentioning

confidence: 99%

Federated Self-training for Semi-supervised Audio Recognition

Tsouvalas

Saeed

Özçelebi

2022

ACM Trans. Embed. Comput. Syst.

View full text Add to dashboard Cite

Federated Learning is a distributed machine learning paradigm dealing with decentralized and personal datasets. Since data reside on devices like smartphones and virtual assistants, labeling is entrusted to the clients or labels are extracted in an automated way. Specifically, in the case of audio data, acquiring semantic annotations can be prohibitively expensive and time-consuming. As a result, an abundance of audio data remains unlabeled and unexploited on users’ devices. Most existing federated learning approaches focus on supervised learning without harnessing the unlabeled data. In this work, we study the problem of semi-supervised learning of audio models via self-training in conjunction with federated learning. We propose FedSTAR to exploit large-scale on-device unlabeled data to improve the generalization of audio recognition models. We further demonstrate that self-supervised pre-trained models can accelerate the training of on-device models, significantly improving convergence within fewer training rounds. We conduct experiments on diverse public audio classification datasets and investigate the performance of our models under varying percentages of labeled and unlabeled data. Notably, we show that with as little as 3% labeled data available, FedSTAR on average can improve the recognition rate by 13.28% compared to the fully-supervised federated model.

show abstract

“…Despite the growing number of studies applying FL on speech-related tasks [19,20,21,22,23], very few of these have investigated its use for end-to-end (E2E) ASR. To our best knowledge, existing works on FL for ASR typically rely on strong simplifying assumptions for many of these challengesand this results in their experimental settings being still far away from the conditions in which a FL ASR would need to function.…”

Section: Introductionmentioning

confidence: 99%

End-to-End Speech Recognition from Federated Acoustic Models

Gao¹,

Parcollet²,

Zaiem³

et al. 2021

Preprint

View full text Add to dashboard Cite

Training Automatic Speech Recognition (ASR) models under federated learning (FL) settings has recently attracted considerable attention. However, the FL scenarios often presented in the literature are artificial and fail to capture the complexity of real FL systems. In this paper, we construct a challenging and realistic ASR federated experimental setup consisting of clients with heterogeneous data distributions using the French Common Voice dataset, a large heterogeneous dataset containing over 10k speakers. We present the first empirical study on attention-based sequence-to-sequence E2E ASR model with three aggregation weighting strategies -standard FedAvg, lossbased aggregation and a novel word error rate (WER)-based aggregation, are conducted in two realistic FL scenarios: crosssilo with 10-clients and cross-device with 2k-clients. In particular, the WER-based weighting method is proposed to better adapt FL to the context of ASR by integrating the error rate metric with the aggregation process. Our analysis on E2E ASR from heterogeneous and realistic federated acoustic models provides the foundations for future research and development of realistic FL-based ASR applications.

show abstract

Training Keyword Spotting Models on Non-IID Data with Federated Learning

Cited by 10 publications

References 21 publications

FLAME: Federated Learning Across Multi-device Environments

FLAME: Federated Learning Across Multi-device Environments

Federated Self-training for Semi-supervised Audio Recognition

End-to-End Speech Recognition from Federated Acoustic Models

Contact Info

Product

Resources

About