Federated Continuous Learning With Broad Network Architecture

Le, Junqing; Lei, Xinyu; Mu, Nankun; Zhang, Hengrun; Zeng, Kai; Liao, Xiaofeng

doi:10.1109/tcyb.2021.3090260

Cited by 38 publications

(15 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that we average the edge-model over all clients while only average the global-model over a subset S t of edge servers, since the communication between the edge servers and clients is efficient, but the communication between the edge servers and the cloud server is high cost and latency because the distance is relatively long. At the client side, i.e., at the third level, the sparse personalized client model θi,j (y t,r i,j ) of the i-th edge server and the j-th client is determined by solving (6), where y t,r i,j is the local edge-model of the i-th edge server and the j-th client at the global round t and edge round r. The sparse client model used here can reduce the communication load between clients and edge servers. Note that ( 6) can be easily solved by many first order approaches, for example the Nesterov's accelerated gradient descent, based on the gradient…”

Section: Sfedhp: Algorithmmentioning

confidence: 99%

“…That is, we solve the following minimization problem instead of solving (6) to obtain an approximated personalized client model…”

Section: Sfedhp: Algorithmmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…It enables secure co-training meeting the requirements for privacy and security concerns, by collecting clients' trained models, instead of local data, to a centralized server to generate the global model. And FL has been used in the Internet of Things (IoT), wireless networks, mobile devices, autonomous vehicles, and human activity recognition gradually [4][5][6][7][8], for its excellent potential in cybersecurity and privacy security.However, the frequency communication between clients and the server is usually required in FL to ensure convergence performance, which has high latency and limited communication bandwidth. Therefore, effective communication methods must be used, such as sparse optimizers [9][10][11].…”

mentioning

confidence: 99%

See 3 more Smart Citations

Sparse Federated Learning with Hierarchical Personalized Models

Liu¹,

Li²,

Shao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Federated learning (FL) is widely used in the Internet of Things (IoT), wireless networks, mobile devices, autonomous vehicles, and human activity due to its excellent potential in cybersecurity and privacy security. Though FL method can achieve privacy-safe and reliable collaborative training without collecting users' privacy data, it suffers from many challenges during both training and deployment. The main challenges in FL are the difficulty of non-i.i.d co-training data caused by the statistical diversity of the data from various participants, and the difficulty of application deployment caused by the excessive traffic volume and long communication delay between the central server and the client. To address these problems, we propose a sparse FL scheme with hierarchical personalization models (sFedHP), which minimizes clients' loss functions including the properties of an approximated 1 -norm and the hierarchical proximal mapping, to reduce the communicational and computational loads required in the network, while improving the performance on statistical diversity data. Convergence analysis shows that the sparse constraint in sFedHP only reduces the convergence speed to a small extent, while the communication cost is greatly reduced. Experimentally, we demonstrate the benefits of this sparse hierarchical personalization architecture compared with the client-edge-cloud hierarchical FedAvg and the state-of-the-art personalization methods. IntroductionMachine learning methods have grown rapidly in a wide range of applications thanks to a large number of labeled training samples [1]. Typically, these samples collected on users' devices, such as mobile phones, are expected to send to a centralized server with powerful computing power to train a deep model [2]. However, users are often reluctant to share personal data due to privacy and security concerns, which motivates the emergence of federated learning (FL) [3]. Federated Averaging (FedAvg) [3] is known as the first FL algorithms to build a global-model in different clients while protecting their personal data locally. It enables secure co-training meeting the requirements for privacy and security concerns, by collecting clients' trained models, instead of local data, to a centralized server to generate the global model. And FL has been used in the Internet of Things (IoT), wireless networks, mobile devices, autonomous vehicles, and human activity recognition gradually [4][5][6][7][8], for its excellent potential in cybersecurity and privacy security.However, the frequency communication between clients and the server is usually required in FL to ensure convergence performance, which has high latency and limited communication bandwidth. Therefore, effective communication methods must be used, such as sparse optimizers [9][10][11]. Afterward, one-shot FL [12] enables a central server learns a global-model in a single round of communication. Quantization methods [13,14] and multiple local optimization rounds [15,16] have been utilized to address the limitations on...

show abstract

Section: Sfedhp: Algorithmmentioning

confidence: 99%

“…That is, we solve the following minimization problem instead of solving (6) to obtain an approximated personalized client model…”

Section: Sfedhp: Algorithmmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Sparse Federated Learning with Hierarchical Personalized Models

Liu¹,

Li²,

Shao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For model developers who prototype a mobile AI with FL without a proxy dataset, achieving faster convergence on thousands to millions of devices is desired to efficiently test multiple model architectures and hyperparameters [29]. Service providers who frequently update a model with continual learning with FL require to minimize the user overhead with better time-to-accuracy performance [35].…”

Section: Introductionmentioning

confidence: 99%

FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clients

Shin¹,

Li²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Federated Learning (FL) trains a machine learning model on distributed clients without exposing individual data. Unlike centralized training that is usually based on carefullyorganized data, FL deals with on-device data that are often unfiltered and imbalanced. As a result, conventional FL training protocol that treats all data equally leads to a waste of local computational resources and slows down the global learning process. To this end, we propose FedBalancer, a systematic FL framework that actively selects clients' training samples. Our sample selection strategy prioritizes more "informative" data while respecting privacy and computational capabilities of clients. To better utilize the sample selection to speed up global training, we further introduce an adaptive deadline control scheme that predicts the optimal deadline for each round with varying client train data. Compared with existing FL algorithms with deadline configuration methods, our evaluation on five datasets from three different domains shows that FedBalancer improves the time-to-accuracy performance by 1.22∼4.62× while improving the model accuracy by 1.0∼3.3%. We also show that FedBalancer is readily applicable to other FL approaches by demonstrating that FedBalancer improves the convergence speed and accuracy when operating jointly with three different FL algorithms.

show abstract