Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementation

Wang, Su; Lee, Mengyuan; Hosseinalipour, Seyyedali; Morabito, Roberto; Chiang, Mung; Brinton, Christopher G.

doi:10.1109/infocom42981.2021.9488906

Cited by 98 publications

(23 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, researchers have studied the model training performance under noisy channels [14], limited energy devices [12], limited bandwidth [13], quantization and sparsification [15], [16], and wireless aggregation of signals over the air [17]. Also, device sampling [18] and data sampling [19] has been topics of research. Furthermore, a part of literature focuses on adapting FedL for a variety of new technologies, such as unmanned aerial vehicles [20], [21], intelligent reflecting surfaces [22], and massive MIMO [23].…”

Section: B Related Workmentioning

confidence: 99%

Multi-Edge Server-Assisted Dynamic Federated Learning with an Optimized Floating Aggregation Point

Bhargav¹,

Hosseinalipour²,

Kim³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We propose cooperative edge-assisted dynamic federated learning (CE-FL). CE-FL introduces a distributed machine learning (ML) architecture, where data collection is carried out at the end devices, while the model training is conducted cooperatively at the end devices and the edge servers, enabled via data offloading from the end devices to the edge servers through base stations. CE-FL also introduces floating aggregation point, where the local models generated at the devices and the servers are aggregated at an edge server, which varies from one model training round to another to cope with the network evolution in terms of data distribution and users' mobility. CE-FL considers the heterogeneity of network elements in terms of communication/computation models and the proximity to one another. CE-FL further presumes a dynamic environment with online variation of data at the network devices which causes a drift at the ML model performance. We model the processes taken during CE-FL, and conduct analytical convergence analysis of its ML model training. We then formulate network-aware CE-FL which aims to adaptively optimize all the network elements via tuning their contribution to the learning process, which turns out to be a non-convex mixed integer problem. Motivated by the large scale of the system, we propose a distributed optimization solver to break down the computation of the solution across the network elements. We finally demonstrate the effectiveness of our framework with the data collected from a real-world testbed.

show abstract

Section: B Related Workmentioning

confidence: 99%

Multi-Edge Server-Assisted Dynamic Federated Learning with an Optimized Floating Aggregation Point

Bhargav¹,

Hosseinalipour²,

Kim³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We impose no constraint on client selection [40,43,45,52,76,81,88] or training data sampling [44,76] strategies, making it compatible with a mass of recent FL system literature.…”

Section: Federated Learningmentioning

confidence: 99%

“…Integration with the existing FL framework AutoFedNLP's trial groups are compatible with how existing FL frameworks manage clients for training efficiency, a key system component having received high research attention [40,44,45,52,76,81,88]. This is because the adapters and their configuration scheduler are intentionally designed to be decoupled from which device or data will be involved in per-round training.…”

Section: Configurator Algorithm In Detailmentioning

confidence: 99%

See 1 more Smart Citation

AutoFedNLP: An efficient FedNLP framework

Cai¹,

Wu²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformer-based pre-trained models have revolutionized NLP for superior performance and generality. Fine-tuning pre-trained models for downstream tasks often requires private data, for which federated learning is the de-facto approach (i.e., FedNLP). However, our measurements show that FedNLP is prohibitively slow due to the large model sizes and the resultant high network/computation cost. Towards practical FedNLP, we identify as the key building blocks adapters, small bottleneck modules inserted at a variety of model layers. A key challenge is to properly configure the depth and width of adapters, to which the training speed and efficiency is highly sensitive. No silver-bullet configuration exists: the optimal choice varies across downstream NLP tasks, desired model accuracy, and client resources. To automate adapter configuration, we propose AutoFedNLP, a framework that enhances the existing FedNLP with two novel designs. First, AutoFedNLP progressively upgrades the adapter configuration throughout a training session; the principle is to quickly learn shallow knowledge by only training fewer and smaller adapters at the model's top layers, and incrementally learn deep knowledge by incorporating deeper and larger adapters. Second, AutoFedNLP continuously profiles future adapter configurations by allocating participant devices to trial groups. To minimize client-side computations, AutoFedNLP exploits the fact that a FedNLP client trains on the same samples repeatedly between consecutive changes of adapter configurations, and caches computed activations on clients. Extensive experiments show that AutoFedNLP can reduce FedNLP's model convergence delay to no more than several hours, which is up to 155.5× faster compared to vanilla FedNLP and 48× faster compared to strong baselines.

show abstract

“…In this paper, we investigate the problem of machine unlearning in a more practical scenario, where data holders are collaboratively performing training and unlearning without sharing raw data. In particular, we target Federated Learning (FL) [11]- [17], a widely adopted privacy-aware collaborative learning framework. In FL, data holders train a model from their local data samples, and the server only aggregates data holders' local model updates for data privacy considerations [18].…”

Section: Introductionmentioning

confidence: 99%

The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retraining

Liu,

Xu,

Yuan

et al. 2022

Preprint

View full text Add to dashboard Cite

In Machine Learning, the emergence of the right to be forgotten gave birth to a paradigm named machine unlearning, which enables data holders to proactively erase their data from a trained model. Existing machine unlearning techniques focus on centralized training, where access to all holders' training data is a must for the server to conduct the unlearning process. It remains largely underexplored about how to achieve unlearning when full access to all training data becomes unavailable. One noteworthy example is Federated Learning (FL), where each participating data holder trains locally, without sharing their training data to the central server. In this paper, we investigate the problem of machine unlearning in FL systems. We start with a formal definition of the unlearning problem in FL and propose a rapid retraining approach to fully erase data samples from a trained FL model. The resulting design allows data holders to jointly conduct the unlearning process efficiently while keeping their training data locally. Our formal convergence and complexity analysis demonstrate that our design can preserve model utility with high efficiency. Extensive evaluations on four real-world datasets illustrate the effectiveness and performance of our proposed realization.

show abstract

Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementation

Cited by 98 publications

References 23 publications

Multi-Edge Server-Assisted Dynamic Federated Learning with an Optimized Floating Aggregation Point

Multi-Edge Server-Assisted Dynamic Federated Learning with an Optimized Floating Aggregation Point

AutoFedNLP: An efficient FedNLP framework

The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retraining

Contact Info

Product

Resources

About