Multi-center federated learning: clients clustering for better personalization

Abstract

Personalized decision-making can be implemented in a Federated learning (FL) framework that can collaboratively train a decision model by extracting knowledge across intelligent clients, e.g. smartphones or enterprises. FL can mitigate the data privacy risk of collaborative training since it merely collects local gradients from users without access to their data. However, FL is fragile in the presence of statistical heterogeneity that is commonly encountered in personalized decision making, e.g., non-IID data over different clients. Existing FL approaches usually update a single global model to capture the shared knowledge of all users by aggregating their gradients, regardless of the discrepancy between their data distributions. By comparison, a mixture of multiple global models could capture the heterogeneity across various clients if assigning the client to different global models (i.e., centers) in FL. To this end, we propose a novel multi-center aggregation mechanism to cluster clients using their models’ parameters. It learns multiple global models from data as the cluster centers, and simultaneously derives the optimal matching between users and centers. We then formulate it as an optimization problem that can be efficiently solved by a stochastic expectation maximization (EM) algorithm. Experiments on multiple benchmark datasets of FL show that our method outperforms several popular baseline methods. The experimental source codes are publicly available on the Github repository (GitHub repository: https://github.com/mingxuts/multi-center-fed-learning).

Combining Federated Learning and Edge Computing toward Ubiquitous Intelligence: Challenges, Recent Advances, and Future Directions

“…The flexible clustered FL framework (FlexCFL) proposed in [33] groups clients based on the similarities between the clients' optimization directions for achieving lower training divergence within clusters. Long et al [34] designed a multi-center federated loss as the objective function for client clustering in FL and proposed federated stochastic expectation maximization (FeSEM) algorithm to optimize this objective.…”

Auxo: Heterogeneity-Mitigating Federated Learning via Scalable Client Clustering

“…Such low availability and partial participation limit the available information for the clustering algorithms. This, however, is ignored by CFL (Sattler et al, 2021), multicenter (Long et al, 2022) and FL+HC (Briggs et al, 2020), and makes the deployment impractical as they requires a complete pass over the entire population to identify clus- ters. Furthermore, clients usually have limited on-board resources, but IFCA (Ghosh et al, 2020), FlexCFL (Duan et al, 2021), and FL+HC require extra computation to for every client and/or over time to assign them to a cluster, which incurs large computation and communication overhead.…”

“…On the one hand, statistical heterogeneity makes one single global model insufficient for satisfying every client's data distributions (Li et al, 2020c;Zhao et al, 2018). Even if a personalization algorithm is used to generate individual models for each client, they may suffer from heterogeneity-borne challenges (Tang et al, 2021;Long et al, 2022). Several studies that try to mitigate the effect of statistical heterogeneity, such as FedYoGi (Reddi et al, 2021), q-FedAvg (Li et al, 2020b), FTFA (Cheng et al, 2021), have shown that their convergence depends on the Under Review degree of heterogeneity, both theoretically and empirically.…”

“…Although some recent works that have attempted to train separate models for different client groups (Briggs et al, 2020;Ghosh et al, 2020;Long et al, 2022), how to cluster clients at scale and in the wild while respecting client privacy has received little systematic attention ( §2.4). To this end, we propose Auxo, a cohort manager for efficient FL, to enable 1) scalable cohort identification to reduce intra-cohort heterogeneity under large-scale and low-participation FL scenarios; and 2) efficient cohort-based training to guarantee model convergence and facilitate most FL optimizations (e.g., fairness optimization) without extra resource cost.…”

A Framework for Multi-Prototype Based Federated Learning: Towards the Edge Intelligence

“…Additionally, [24] explores a benchmark for non-IID settings, they divide non-IID settings into five cases, such as label distribution skew, feature distribution skew, quantity skew, etc. Further, as [24] mentioned, some existing studies [15]- [17], [25] cover only one non-IID case, which do not give sufficient evaluations to this challenge. Therefore, to avoid the influence of biased global models and to evaluate non-IID cases as comprehensively as possible, we focus on personalized FL by optimizing the local objective of each local client under the label and feature distribution skewness.…”