Fast-Convergent Federated Learning with Adaptive Weighting

Wu, Hongda; Wang, Ping

doi:10.1109/icc42927.2021.9500890

Cited by 18 publications

(25 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Providing the variety of different nodes on contributing global model, to improve the convergence rate, one can seek to preferentially select the nodes with higher contribution (i.e., the nodes with i.i.d. dataset, as observed in [6], [18]). As such, we propose a probabilistic node selection design, which is an unbiased node selection scheme that can dynamically change the probability for each node to be selected in each communication round, based on their data distribution-related contribution, which can be distinguished by the procedure CHECK EXPECTATION in Optimal Aggregation.…”

Section: Fl With Probabilistic Node Selection (Fedpns)mentioning

confidence: 65%

“…data distribution across participating nodes, aiming to reduce communication rounds in FL. These studies include adaptive tuning local training [9], weighting design for model aggregation [18], and node selection strategies [19]- [22]. The algorithm FedProx by Li et al [9] uses a regularization term to balance between the optimizing discrepancy of global-local objectives and allowing participating nodes to perform a variable number of local updates, to consequently overcome the non-i.i.d.…”

Section: Related Workmentioning

confidence: 99%

“…data distribution and resource heterogeneity. Authors in [18] exhibited a contribution-related weighting design, namely FedAdp to boost FL convergence rate in the presence of nodes with non-i.i.d. data samples, which assigns distinguished weight for participating nodes according to the correlation between local objective and global objective revealed by gradient information.…”

Section: Related Workmentioning

confidence: 99%

“…Though FedAvg can achieve a decent convergence rate with trivial node selection policy and simple averaging design, partial node participation and non-i.i.d. training data slows down the convergence rate [13], which is also observed in [10]- [12], [18], [22]. Since communication cost becomes a critical bottleneck in FL, one can increase the local computing (i.e., more local updates), which is shown to be beneficial to save communication rounds and improve the convergence rate [6], [12].…”

Section: The Challenges Due To Non-iid Data Distributionmentioning

confidence: 99%

See 3 more Smart Citations

Node Selection Toward Faster Convergence for Federated Learning on Non-IID Data

Wu¹,

Wang²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Federated Learning (FL) is a distributed learning paradigm that enables a large number of resource-limited nodes to collaboratively train a model without data sharing. The non-independent-and-identically-distributed (non-i.i.d.) data samples invoke discrepancy between global and local objectives, making the FL model slow to converge. In this paper, we proposed Optimal Aggregation algorithm for better aggregation, which finds out the optimal subset of local updates of participating nodes in each global round, by identifying and excluding the adverse local updates via checking the relationship between the local gradient and the global gradient. Then, we proposed a Probabilistic Node Selection framework (FedPNS) to dynamically change the probability for each node to be selected based on the output of Optimal Aggregation. FedPNS can preferentially select nodes that propel faster model convergence. The unbiasedness of the proposed FedPNS design is illustrated and the convergence rate improvement of FedPNS over the commonly adopted Federated Averaging (FedAvg) algorithm is analyzed theoretically. Experimental results demonstrate the effectiveness of FedPNS in accelerating the FL convergence rate, as compared to FedAvg with random node selection.

show abstract

Section: Fl With Probabilistic Node Selection (Fedpns)mentioning

confidence: 65%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: The Challenges Due To Non-iid Data Distributionmentioning

confidence: 99%

See 2 more Smart Citations

Node Selection Toward Faster Convergence for Federated Learning on Non-IID Data

Wu¹,

Wang²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Meanwhile, the influence of malicious models should be lowered as well. We propose using the weighted federated averaging method [16], a modified version of FedAvg, in this paper. The modified FedAvg mechanism works together with the reputation system to keep the federated ML secured and stable.…”

Section: Weighted Federated Averagingmentioning

confidence: 99%

DFL: High-Performance Blockchain-Based Federated Learning

Tian¹,

Guo²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Many researchers are trying to replace the aggregation server in federated learning with a blockchain system to achieve better privacy, robustness and scalability. In this case, clients will upload their updated models to the blockchain ledger, and use a smart contract on the blockchain system to perform model averaging. However, running machine learning applications on the blockchain is almost impossible because a blockchain system, which usually takes over half minute to generate a block, is extremely slow and unable to support machine learning applications.This paper proposes a completely new public blockchain architecture called DFL, which is specially optimized for distributed federated machine learning. This architecture inherits most traditional blockchain merits and achieves extremely high performance with low resource consumption by waiving global consensus. To characterize the performance and robustness of our architecture, we implement the architecture as a prototype and test it on a physical four-node network. To test more nodes and more complex situations, we build a simulator to simulate the network. The LeNet results indicate our system can reach over 90% accuracy for non-I.I.D. datasets even while facing model poisoning attacks, with the blockchain consuming less than 5% of hardware resources.

show abstract

Federated learning with stochastic quantization

Xue

2022

Int J of Intelligent Sys

View full text Add to dashboard Cite

This paper studies the distributed federated learning problem when the exchanged information between the server and the workers is quantized. A novel quantized federated averaging algorithm is developed by applying stochastic quantization scheme to the local and global model parameters. Specifically, the server broadcasts the quantized global model parameter to the workers; the workers update local model parameters using their own data sets and upload the quantized version to the server; then the server updates the global model parameter by aggregating all the quantized local model parameters and its previous global model parameter. This algorithm can be interpreted as a quantized variant of the federated averaging algorithm. The convergence is analyzed theoretically for both convex and strongly convex loss functions with Lipschitz gradient. Extensive experiments using realistic data are provided to show the effectiveness of the proposed algorithm.

show abstract

Fast-Convergent Federated Learning with Adaptive Weighting

Cited by 18 publications

References 8 publications

Node Selection Toward Faster Convergence for Federated Learning on Non-IID Data

Node Selection Toward Faster Convergence for Federated Learning on Non-IID Data

DFL: High-Performance Blockchain-Based Federated Learning

Federated learning with stochastic quantization

Contact Info

Product

Resources

About