Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learning

Bouacida, Nader; Hou, Jiahui; Zang, Hui; Liu, Xin

doi:10.48550/arxiv.2011.04050

Cited by 5 publications

(7 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have been other FL studies inspired by the dropout method [28], [29]. Namely, "Federated Dropout" [28] and "Adaptive Federated Dropout" [29].…”

Section: Discussion and Future Directionsmentioning

confidence: 99%

“…There have been other FL studies inspired by the dropout method [28], [29]. Namely, "Federated Dropout" [28] and "Adaptive Federated Dropout" [29]. The main goal of both of these approaches is to increase communication efficiency by decreasing the model size to be sent and received by the local clients(mobile devices).…”

Section: Discussion and Future Directionsmentioning

confidence: 99%

“…In these approaches, each local client trains a smaller model (a submodel of the global model) to reduce model size and update size, while the server has the whole global model. Clients train a selected subset of the global model, and it is either a random subset [28] or dynamically selected subset [29]. Then, the server maps and reunites those smaller locally trained models into the global model.…”

Section: Discussion and Future Directionsmentioning

confidence: 99%

See 2 more Smart Citations

FedDropoutAvg: Generalizable federated learning for histopathology image classification

Güneşli¹,

Bilal²,

Raza³

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated learning (FL) enables collaborative learning of a deep learning model without sharing the data of participating sites. FL in medical image analysis tasks is relatively new and open for enhancements. In this study, we propose FedDropoutAvg, a new federated learning approach for training a generalizable model. The proposed method takes advantage of randomness, both in client selection and also in federated averaging process. We compare FedDropoutAvg to several algorithms in an FL scenario for real-world multi-site histopathology image classification task. We show that with FedDropoutAvg, the final model can achieve performance better than other FL approaches and closer to a classical deep learning model that requires all data to be shared for centralized training. We test the trained models on a large dataset consisting of 1.2 million image tiles from 21 different centers. To evaluate the generalization ability of the proposed approach, we use held-out test sets from centers whose data was used in the FL and for unseen data from other independent centers whose data was not used in the federated training. We show that the proposed approach is more generalizable than other state-of-the-art federated training approaches. To the best of our knowledge, ours is the first study to use a randomized client and local model parameter selection procedure in a federated setting for a medical image analysis task.

show abstract

“…There have been other FL studies inspired by the dropout method [28], [29]. Namely, "Federated Dropout" [28] and "Adaptive Federated Dropout" [29].…”

Section: Discussion and Future Directionsmentioning

confidence: 99%

Section: Discussion and Future Directionsmentioning

confidence: 99%

Section: Discussion and Future Directionsmentioning

confidence: 99%

See 1 more Smart Citation

FedDropoutAvg: Generalizable federated learning for histopathology image classification

Güneşli¹,

Bilal²,

Raza³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Due to the aforementioned superiorities, AQG can be used jointly with some other communication efficient methods for FL architectures, such as gradient sparsification [23], client selection based on local resources [13,20,29] and adaptively distributing subnetworks for heterogeneous clients [6,9]. Such superiorities and flexibility endow great potentials for the proposed FL framework with AQG.…”

Section: Discussionmentioning

confidence: 99%

Communication Efficient Federated Learning with Adaptive Quantization

Mao¹,

Zhao²,

Yan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated learning (FL) has attracted tremendous attentions in recent years due to its privacy preserving measures and great potentials in some distributed but privacy-sensitive applications like finance and health. However, high communication overloads for transmitting high-dimensional networks and extra security masks remains a bottleneck of FL. This paper proposes a communication-efficient FL framework with Adaptive Quantized Gradient (AQG) which adaptively adjusts the quantization level based on local gradient's update to fully utilize the heterogeneousness of local data distribution for reducing unnecessary transmissions. Besides, the client dropout issues are taken into account and the Augmented AQG is developed, which could limit the dropout noise with an appropriate amplification mechanism for transmitted gradients. Theoretical analysis and experiment results show that the proposed AQG leads to 25%-50% of additional transmission reduction as compared to existing popular methods including Quantized Gradient Descent (QGD) and Lazily Aggregated Quantized (LAQ) gradient-based method without deteriorating convergence properties. Particularly, experiments with heterogenous data distributions corroborate a more significant transmission reduction compared with independent identical data distributions. Meanwhile, the proposed AQG is robust to a client dropping rate up to 90% empirically, and the Augmented AQG manages to further improve the FL system's communication efficiency with the presence of moderate-scale client dropouts commonly seen in practical FL scenarios.

show abstract

“…Recently, compression-based methods have been widely adopted to improve the communication efficiency of FL, where only a part of the weight or gradient information is transmitted. A dropout approach was considered in [4], where a partial network is dropped during the training to reduce the number of parameters to be transmitted. Similarly, Section II, we introduce the general FL system and the proposed FedDLR method.…”

Section: Introductionmentioning

confidence: 99%

Communication-Efficient Federated Learning with Dual-Side Low-Rank Compression

Qiao¹,

Yu²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated learning (FL) is a promising and powerful approach for training deep learning models without sharing the raw data of clients. During the training process of FL, the central server and distributed clients need to exchange a vast amount of model information periodically. To address the challenge of communication-intensive training, we propose a new training method, referred to as federated learning with dualside low-rank compression (FedDLR), where the deep learning model is compressed via low-rank approximations at both the server and client sides. The proposed FedDLR not only reduces the communication overhead during the training stage but also directly generates a compact model to speed up the inference process. We shall provide convergence analysis, investigate the influence of the key parameters, and empirically show that FedDLR outperforms the state-of-the-art solutions in terms of both the communication and computation efficiency.

show abstract

Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learning

Cited by 5 publications

References 5 publications

FedDropoutAvg: Generalizable federated learning for histopathology image classification

FedDropoutAvg: Generalizable federated learning for histopathology image classification

Communication Efficient Federated Learning with Adaptive Quantization

Communication-Efficient Federated Learning with Dual-Side Low-Rank Compression

Contact Info

Product

Resources

About