Communication-Efficient Learning of Deep Networks from Decentralized Data

McMahan, H. Brendan; Moore, Eider B; Ramage, Daniel; Hampson, Seth; Arcas, Blaise Agüera y

doi:10.48550/arxiv.1602.05629

Cited by 448 publications

(717 citation statements)

References 16 publications

Supporting

Mentioning

713

Contrasting

Order By: Relevance

“…Baselines. We compare our proposed FedSpa with four baselines, including FedAvg [6], Sub-FedAvg [5], Ditto [3] and Local. We tune the hyper-parameters of the baselines to their best states.…”

Section: Methodsmentioning

confidence: 99%

“…The learning rate is initialized with 0.1 and decayed with 0.998 after each communication round. We simulate 100 clients in total, and in each round 10 of them are picked to perform local training (the setting follows [6]). For all the methods except Ditto, local epochs are fixed to 5.…”

Section: Methodsmentioning

confidence: 99%

“…Federated learning (FL) [6] is seriously afflicted by the issue of heterogeneously distributed (or Non-IID) data. Personalized FL (PFL), initiated by recent literature [2], [3], is shown to be effective to counter this issue of FL.…”

Section: Related Workmentioning

confidence: 99%

“…• FedAvg [6] is the vanilla solution of FL. It utilizes weights average to enable all the clients to collaboratively train a global model, which efficiently absorbs knowledge from personal data resided in clients.…”

Section: B3 Baseline Descriptionmentioning

confidence: 99%

“…Theoretically, we conclude that the proposed solution can achieve sub-linear convergence towards the formulated problem. Empirically, in the Non-IID setting, we demonstrate that FedSpa accelerates the convergence (respectively 76.2% and 38.1% less communication rounds to reach the best accuracy of FedAvg [6] and Ditto [3]), increases the final accuracy (up to 21.9% and 4.4% higher accuracy than FedAvg and Ditto, respectively), reduces the communication overhead (50% less parame-1. Sparsity specifies the ratio of parameters that are set to 0 (or inactive) in a model.…”

Section: Introductionmentioning

confidence: 95%

See 4 more Smart Citations

Achieving Personalized Federated Learning with Sparse Local Models

Huang¹,

Liu²,

Shen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Federated learning (FL) is vulnerable to heterogeneously distributed data, since a common global model in FL may not adapt to the heterogeneous data distribution of each user. To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user. However, PFL is far from its maturity, because existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory. In this work, we propose federated learning with personalized sparse mask (FedSpa), a novel PFL scheme that employs personalized sparse masks to customize sparse local models on the edge. Instead of training an intact (or dense) PFL model, FedSpa only maintains a fixed number of active parameters throughout training (aka sparse-to-sparse training), which enables users' models to achieve personalization with cheap communication, computation, and memory cost. We theoretically show that the iterates obtained by FedSpa converge to the local minimizer of the formulated SPFL problem at the rate of O( 1 √ T). Comprehensive experiments demonstrate that FedSpa significantly saves communication and computation costs, while simultaneously achieves higher model accuracy and faster convergence speed against several state-of-the-art PFL methods.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: B3 Baseline Descriptionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 95%

See 3 more Smart Citations

Achieving Personalized Federated Learning with Sparse Local Models

Huang¹,

Liu²,

Shen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Federated Learning for Decentralized Artificial Intelligence in Melanoma Diagnostics

Haggenmüller,

Schmitt,

Krieghoff-Henning

et al. 2024

JAMA Dermatol

View full text Add to dashboard Cite

ImportanceThe development of artificial intelligence (AI)–based melanoma classifiers typically calls for large, centralized datasets, requiring hospitals to give away their patient data, which raises serious privacy concerns. To address this concern, decentralized federated learning has been proposed, where classifier development is distributed across hospitals.ObjectiveTo investigate whether a more privacy-preserving federated learning approach can achieve comparable diagnostic performance to a classical centralized (ie, single-model) and ensemble learning approach for AI-based melanoma diagnostics.Design, Setting, and ParticipantsThis multicentric, single-arm diagnostic study developed a federated model for melanoma-nevus classification using histopathological whole-slide images prospectively acquired at 6 German university hospitals between April 2021 and February 2023 and benchmarked it using both a holdout and an external test dataset. Data analysis was performed from February to April 2023.ExposuresAll whole-slide images were retrospectively analyzed by an AI-based classifier without influencing routine clinical care.Main Outcomes and MeasuresThe area under the receiver operating characteristic curve (AUROC) served as the primary end point for evaluating the diagnostic performance. Secondary end points included balanced accuracy, sensitivity, and specificity.ResultsThe study included 1025 whole-slide images of clinically melanoma-suspicious skin lesions from 923 patients, consisting of 388 histopathologically confirmed invasive melanomas and 637 nevi. The median (range) age at diagnosis was 58 (18-95) years for the training set, 57 (18-93) years for the holdout test dataset, and 61 (18-95) years for the external test dataset; the median (range) Breslow thickness was 0.70 (0.10-34.00) mm, 0.70 (0.20-14.40) mm, and 0.80 (0.30-20.00) mm, respectively. The federated approach (0.8579; 95% CI, 0.7693-0.9299) performed significantly worse than the classical centralized approach (0.9024; 95% CI, 0.8379-0.9565) in terms of AUROC on a holdout test dataset (pairwise Wilcoxon signed-rank, P &lt; .001) but performed significantly better (0.9126; 95% CI, 0.8810-0.9412) than the classical centralized approach (0.9045; 95% CI, 0.8701-0.9331) on an external test dataset (pairwise Wilcoxon signed-rank, P &lt; .001). Notably, the federated approach performed significantly worse than the ensemble approach on both the holdout (0.8867; 95% CI, 0.8103-0.9481) and external test dataset (0.9227; 95% CI, 0.8941-0.9479).Conclusions and RelevanceThe findings of this diagnostic study suggest that federated learning is a viable approach for the binary classification of invasive melanomas and nevi on a clinically representative distributed dataset. Federated learning can improve privacy protection in AI-based melanoma diagnostics while simultaneously promoting collaboration across institutions and countries. Moreover, it may have the potential to be extended to other image classification tasks in digital cancer histopathology and beyond.

show abstract

From distributed machine learning to federated learning: In the view of data privacy and security

Shen

Zhu

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

Federated learning is an improved version of distributed machine learning that further offloads operations which would usually be performed by a central server. The server becomes more like an assistant coordinating clients to work together rather than micromanaging the workforce as in traditional DML. One of the greatest advantages of federated learning is the additional privacy and security guarantees it affords.Federated learning architecture relies on smart devices, such as smartphones and IoT sensors, that collect and process their own data, so sensitive information never has to leave the client device. Rather, clients train a submodel locally and send an encrypted update to the central server for aggregation into the global model. These strong privacy guarantees make federated learning an attractive choice in a world where data breaches and information theft are common and serious threats. This survey outlines the landscape and latest developments in data privacy and security for federated learning. We identify the different mechanisms used to provide privacy and security, such as differential privacy, secure multiparty computation and secure aggregation.We also survey the current attack models, identifying the areas of vulnerability and the strategies adversaries use to penetrate federated systems. The survey concludes with a discussion on the open challenges and potential directions of future work in this increasingly popular learning paradigm.

show abstract

Communication-Efficient Learning of Deep Networks from Decentralized Data

Cited by 448 publications

References 16 publications

Achieving Personalized Federated Learning with Sparse Local Models

Achieving Personalized Federated Learning with Sparse Local Models

Federated Learning for Decentralized Artificial Intelligence in Melanoma Diagnostics

From distributed machine learning to federated learning: In the view of data privacy and security

Contact Info

Product

Resources

About