Federated Learning with GAN-Based Data Synthesis for Non-IID Clients

Li, Zijian; Shao, Jiawei; Mao, Yuyi; Wang, Jessie Hui

doi:10.1007/978-3-031-28996-5_2

Cited by 18 publications

(11 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More Discussions for Our Assumptions. Assumptions 1, 2, 3 are standard in nonconvex optimization (Reddi et al, 2016) and FL optimization (Reddi et al, 2021;Li et al, 2020b;2021a)…”

Section: A1 Additional Discussion and Theoretical Resultsmentioning

confidence: 99%

“…This lack of research is likely because: ensemble distillation loss (e.g., R KD in (Global Obj)) is calculated upon local models' averaged logits (ensemble of teachers), so the loss w.r.t each local model (individual teacher) is not directly reflected. However, in standard federated optimization, the "global objective" can usually be "decomposed" into "local objectives" (Li et al, 2020b;a;T Dinh et al, 2020;Fallah et al, 2020), and it is easier to first measure the optimization error on each local objective (i.e., individual teacher) and then derive its implications on the global objective (i.e., ensemble of teachers). Therefore, to make KD loss mathematically tractable for convergence analysis, we focus on its upper bound, i.e., the average of individual distillation loss calculated upon each teacher model.…”

Section: Challenges Of Ensemble Distillation and A Relaxedmentioning

confidence: 99%

“…These are common assumptions used when proving the convergence for non-convex FL optimization (Reddi et al, 2021;Li et al, 2020b;2021a;Ozkara et al, 2021), and we refer the readers to Appendix A.1 for detailed discussions.…”

Section: Assumptionsmentioning

confidence: 99%

“…FL was initially designed to train a single global model for all clients. However, such a one-modelfits-all paradigm is not effective when there is client heterogeneity, i.e., the local data are non-IID across clients with heterogeneous features or label distributions (Li et al, 2020b). Personalized Federated Learning (pFL) (Mansour et al, 2020) has emerged as an effective solution to tackle client heterogeneity.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

PerAda: Parameter-Efficient and Generalizable Federated Learning Personalization with Guarantees

Xie¹,

Huang²,

Wenda³

et al. 2023

Preprint

View full text Add to dashboard Cite

Personalized Federated Learning (pFL) has emerged as a promising solution to tackle data heterogeneity across clients in FL. However, existing pFL methods either (1) introduce high communication and computation costs or (2) overfit to local data, which can be limited in scope, and are vulnerable to evolved test samples with natural shifts. In this paper, we propose PERADA, a parameter-efficient pFL framework that reduces communication and computational costs and exhibits superior generalization performance, especially under test-time distribution shifts. PERADA reduces the costs by leveraging the power of pretrained models and only updates and communicates a small number of additional parameters from adapters. PERADA has good generalization since it regularizes each client's personalized adapter with a global adapter, while the global adapter uses knowledge distillation to aggregate generalized information from all clients. Theoretically, we provide generalization bounds to explain why PERADA improves generalization, and we prove its convergence to stationary points under non-convex settings. Empirically, PERADA demonstrates competitive personalized performance (+4.85% on CheXpert) and enables better out-of-distribution generalization (+5.23% on CIFAR-10-C) on different datasets across natural and medical domains compared with baselines, while only updating 12.6% of parameters per model based on the adapter.

show abstract

“…More Discussions for Our Assumptions. Assumptions 1, 2, 3 are standard in nonconvex optimization (Reddi et al, 2016) and FL optimization (Reddi et al, 2021;Li et al, 2020b;2021a)…”

Section: A1 Additional Discussion and Theoretical Resultsmentioning

confidence: 99%

Section: Challenges Of Ensemble Distillation and A Relaxedmentioning

confidence: 99%

Section: Assumptionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

PerAda: Parameter-Efficient and Generalizable Federated Learning Personalization with Guarantees

Xie¹,

Huang²,

Wenda³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…3 shows that equal averaging over all participants, as provided in FedAvg, achieves the poorest performance as the environment is significantly non-i.i.d. [27]. Meanwhile, FedAvg+ compensates for the overfitting loss from FedAvg.…”

Section: Personalized Decentralized Learning With Knowledge Distillationmentioning

confidence: 99%

Personalized Decentralized Federated Learning with Knowledge Distillation

Jeong¹,

Kountouris²

2023

Preprint

View full text Add to dashboard Cite

Personalization in federated learning (FL) functions as a coordinator for clients with high variance in data or behavior. Ensuring the convergence of these clients' models relies on how closely users collaborate with those with similar patterns or preferences. However, it is generally challenging to quantify similarity under limited knowledge about other users' models given to users in a decentralized network. To cope with this issue, we propose a personalized and fully decentralized FL algorithm, leveraging knowledge distillation techniques to empower each device so as to discern statistical distances between local models. Each client device can enhance its performance without sharing local data by estimating the similarity between two intermediate outputs from feeding local samples as in knowledge distillation. Our empirical studies demonstrate that the proposed algorithm improves the test accuracy of clients in fewer iterations under highly non-independent and identically distributed (non-i.i.d.) data distributions and is beneficial to agents with small datasets, even without the need for a central server.

show abstract