Cluster-driven Graph Federated Learning over Multiple Domains

Caldarola, Debora; Mancini, Massimiliano; Galasso, Fabio; Ciccone, Marco Matteo; Rodolà, Emanuele; Caputo, Barbara

doi:10.1109/cvprw53098.2021.00309

Cited by 64 publications

(26 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, averaging gradients collected from clients having access to a limited subset of tasks may translate in oscillations of the global model, and suboptimal performance on the global distribution [53]. Therefore another line of research looks at improving the aggregation stage using server-side momentum [30] and adaptive optimizers [62], or aggregating task-specific parameters [69,9,10]. To the best of our knowledge, no prior work has attempted to explain the behavior of the model in federated scenarios by looking at the loss surface and convergence minima, which is, in our opinion, a fundamental perspective to fully understand the reasons behind the degradation of heterogeneous performance relative to centralized and homogeneous settings.…”

Section: Statistical Heterogeneity In Federated Learningmentioning

confidence: 99%

“…Federated Training Converges to Sharp Minima. Many works tried to account for this difficulty arising in federated scenarios by enforcing regularization in local optimization so that the local model is not led too far apart from the global one [49,38,31,1,47], or by using momentum on the server-side [30], or learning task-specific parameters keeping distinct models on the server-side [69,9,10]. To the best of our knowledge, this is the first work analyzing and addressing such behavior by looking at the loss landscape.…”

Section: Where Heterogeneous Fl Fails At Generalizingmentioning

confidence: 99%

See 1 more Smart Citation

Improving Generalization in Federated Learning by Seeking Flat Minima

Caldarola¹,

Caputo²,

Ciccone³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Models trained in federated settings often suffer from degraded performances and fail at generalizing, especially when facing heterogeneous scenarios. In this work, we investigate such behavior through the lens of geometry of the loss and Hessian eigenspectrum, linking the model's lack of generalization capacity to the sharpness of the solution. Motivated by prior studies connecting the sharpness of the loss surface and the generalization gap, we show that i) training clients locally with Sharpness-Aware Minimization (SAM) or its adaptive version (ASAM) and ii) averaging stochastic weights (SWA) on the server-side can substantially improve generalization in Federated Learning and help bridging the gap with centralized models. By seeking parameters in neighborhoods having uniform low loss, the model converges towards flatter minima and its generalization significantly improves in both homogeneous and heterogeneous scenarios. Empirical results demonstrate the effectiveness of those optimizers across a variety of benchmark vision datasets (e.g. Cifar10/100, Landmarks-User-160k, Idda) and tasks (large scale classification, semantic segmentation, domain generalization).

show abstract

Section: Statistical Heterogeneity In Federated Learningmentioning

confidence: 99%

Section: Where Heterogeneous Fl Fails At Generalizingmentioning

confidence: 99%

Improving Generalization in Federated Learning by Seeking Flat Minima

Caldarola¹,

Caputo²,

Ciccone³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Following this direction, FedRobust [34] addresses the affine distribution shift by learning affine transformations. Another consistent line of research tackles such issue by learning domain-specific batch normalization (BN) statistics [15], [35], [16]. In particular, SiloBN [15] keeps all BN statistics strictly local and leverages Adaptive Batch Normalization [36] to address new domains at test time, while FedBN [16] introduces local BN layers.…”

Section: B Addressing Statistical Heterogeneity In Flmentioning

confidence: 99%

FedDrive: Generalizing Federated Learning to Semantic Segmentation in Autonomous Driving

Fantauzzo¹,

Fani'²,

Caldarola³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Semantic Segmentation is essential to make selfdriving vehicles autonomous, enabling them to understand their surroundings by assigning individual pixels to known categories. However, it operates on sensible data collected from the users' cars; thus, protecting the clients' privacy becomes a primary concern. For similar reasons, Federated Learning has been recently introduced as a new machine learning paradigm aiming to learn a global model while preserving privacy and leveraging data on millions of remote devices. Despite several efforts on this topic, no work has explicitly addressed the challenges of federated learning in semantic segmentation for driving so far. To fill this gap, we propose FedDrive, a new benchmark consisting of three settings and two datasets, incorporating the real-world challenges of statistical heterogeneity and domain generalization. We benchmark state-of-the-art algorithms from the federated learning literature through an in-depth analysis, combining them with style transfer methods to improve their generalization ability. We demonstrate that correctly handling normalization statistics is crucial to deal with the aforementioned challenges. Furthermore, style transfer improves performance when dealing with significant appearance shifts. We plan to make both the code and the benchmark publicly available to the research community.

show abstract

“…Numerous research papers have addressed data heterogeneity (i.e. non-IID data among local clients) in FL [1,7,13,23,31,39,41], such as improve client sampling fairness [27], adaptive optimization [9,28,37,38], and correct the local updation [16,20,33]. Also, federated learning had been extended in real life applications [8,24].…”

Section: Federated Learningmentioning

confidence: 99%

SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learning

Yu¹,

Nguyen²,

Abebe³

et al. 2021

Preprint

View full text Add to dashboard Cite

Efficient federated learning is one of the key challenges for training and deploying AI models on edge devices. However, maintaining data privacy in federated learning raises several challenges including data heterogeneity, expensive communication cost, and limited resources. In this paper, we address the above issues by (a) introducing a salient parameter selection agent based on deep reinforcement learning on local clients, and aggregating the selected salient parameters on the central server, and (b) splitting a normal deep learning model (e.g., CNNs) as a shared encoder and a local predictor, and training the shared encoder through federated learning while transferring its knowledge to Non-IID clients by the local customized predictor. The proposed method (a) significantly reduces the communication overhead of federated learning and accelerates the model inference, while method (b) addresses the data heterogeneity issue in federated learning. Additionally, we leverage the gradient control mechanism to correct the gradient heterogeneity among clients. This makes the training process more stable and converge faster. The experiments show our approach yields a stable training process and achieves notable results compared with the state-of-the-art methods. Our approach significantly reduces the communication cost by up to 108 GB when training VGG-11, and needed 7.6× less communication overhead when training ResNet-20, while accelerating the local inference by reducing up to 39.7% FLOPs on VGG-11.

show abstract

Cluster-driven Graph Federated Learning over Multiple Domains

Cited by 64 publications

References 13 publications

Improving Generalization in Federated Learning by Seeking Flat Minima

Improving Generalization in Federated Learning by Seeking Flat Minima

FedDrive: Generalizing Federated Learning to Semantic Segmentation in Autonomous Driving

SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learning

Contact Info

Product

Resources

About