Sample-level Data Selection for Federated Learning

Li, Anran; Zhang, Lan; Tan, Juntao; Qin, Yaxuan; Wang, Junhao; Li, Xiangyang

doi:10.1109/infocom42981.2021.9488723

Cited by 86 publications

(19 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that FedBalancer uses the loss of a sample to measure the statistical utility (and thus the importance) of a sample to the current model, similar to Importance Sampling [45,55]. While other studies have also leveraged gradient norm or gradient norm upper bound [3,30,38] to achieve the same goal, we use loss as it is more widely applicable to FL tasks with non gradient-based optimizations [53].…”

Section: Client Sample Selectionmentioning

confidence: 99%

“…Moreover, it requires an example dataset, which is hardly applicable at FL scenarios where the client data distributions are usually unknown. Li et al [38] proposes how we can prioritize samples with higher importance in FL using gradient norm upper bound [30]. However, their approach does not provide how many samples should be selected per each round.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clients

Shin¹,

Li²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Federated Learning (FL) trains a machine learning model on distributed clients without exposing individual data. Unlike centralized training that is usually based on carefullyorganized data, FL deals with on-device data that are often unfiltered and imbalanced. As a result, conventional FL training protocol that treats all data equally leads to a waste of local computational resources and slows down the global learning process. To this end, we propose FedBalancer, a systematic FL framework that actively selects clients' training samples. Our sample selection strategy prioritizes more "informative" data while respecting privacy and computational capabilities of clients. To better utilize the sample selection to speed up global training, we further introduce an adaptive deadline control scheme that predicts the optimal deadline for each round with varying client train data. Compared with existing FL algorithms with deadline configuration methods, our evaluation on five datasets from three different domains shows that FedBalancer improves the time-to-accuracy performance by 1.22∼4.62× while improving the model accuracy by 1.0∼3.3%. We also show that FedBalancer is readily applicable to other FL approaches by demonstrating that FedBalancer improves the convergence speed and accuracy when operating jointly with three different FL algorithms.

show abstract

Section: Client Sample Selectionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clients

Shin¹,

Li²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We impose no constraint on client selection [40,43,45,52,76,81,88] or training data sampling [44,76] strategies, making it compatible with a mass of recent FL system literature.…”

Section: Federated Learningmentioning

confidence: 99%

“…Integration with the existing FL framework AutoFedNLP's trial groups are compatible with how existing FL frameworks manage clients for training efficiency, a key system component having received high research attention [40,44,45,52,76,81,88]. This is because the adapters and their configuration scheduler are intentionally designed to be decoupled from which device or data will be involved in per-round training.…”

Section: Configurator Algorithm In Detailmentioning

confidence: 99%

See 1 more Smart Citation

AutoFedNLP: An efficient FedNLP framework

Cai¹,

Wu²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformer-based pre-trained models have revolutionized NLP for superior performance and generality. Fine-tuning pre-trained models for downstream tasks often requires private data, for which federated learning is the de-facto approach (i.e., FedNLP). However, our measurements show that FedNLP is prohibitively slow due to the large model sizes and the resultant high network/computation cost. Towards practical FedNLP, we identify as the key building blocks adapters, small bottleneck modules inserted at a variety of model layers. A key challenge is to properly configure the depth and width of adapters, to which the training speed and efficiency is highly sensitive. No silver-bullet configuration exists: the optimal choice varies across downstream NLP tasks, desired model accuracy, and client resources. To automate adapter configuration, we propose AutoFedNLP, a framework that enhances the existing FedNLP with two novel designs. First, AutoFedNLP progressively upgrades the adapter configuration throughout a training session; the principle is to quickly learn shallow knowledge by only training fewer and smaller adapters at the model's top layers, and incrementally learn deep knowledge by incorporating deeper and larger adapters. Second, AutoFedNLP continuously profiles future adapter configurations by allocating participant devices to trial groups. To minimize client-side computations, AutoFedNLP exploits the fact that a FedNLP client trains on the same samples repeatedly between consecutive changes of adapter configurations, and caches computed activations on clients. Extensive experiments show that AutoFedNLP can reduce FedNLP's model convergence delay to no more than several hours, which is up to 155.5× faster compared to vanilla FedNLP and 48× faster compared to strong baselines.

show abstract