FedMM: Saddle Point Optimization for Federated Adversarial Domain Adaptation

Abstract

Federated adversary domain adaptation is a unique distributed minimax training task due to the prevalence of label imbalance among clients, with each client only seeing a subset of the classes of labels required to train a global model. To tackle this problem, we propose a distributed minimax optimizer referred to as FedMM, designed specifically for the federated adversary domain adaptation problem. It works well even in the extreme case where each client has different label classes and some clients only have unsupervised tasks. We prove that FedMM ensures convergence to a stationary point with domain-shifted unsupervised data. On a variety of benchmark datasets, extensive experiments show that FedMM consistently achieves either significant communication savings or significant accuracy improvements over federated optimizers based on the gradient descent ascent (GDA) algorithm. When training from scratch, for example, it outperforms other GDA based federated average methods by around 20% in accuracy over the same communication rounds; and it consistently outperforms when training from pre-trained models with an accuracy improvement from 5.4% to 9% for different networks.

FedConceptEM: Robust Federated Learning Under Diverse Distribution Shifts

“…Addressing distribution shifts is a key problem in FL, while most existing works focusing on label distribution skew through techniques such as training robust global models (Li et al, 2018c(Li et al, , 2021 or variance reduction methods (Karimireddy et al, 2020b,a). As another line of research, studies about feature distribution skew in FL mostly focus on domain generalization to train models that can generalize to unseen feature distributions (Peng et al, 2019;Wang et al, 2022a;Shen et al, 2021;Sun et al, 2022;Gan et al, 2021). All of the above methods aim to train a single robust model.…”

“…Many studies focus on adapting DG algorithms for FL scenarios. For example, combining FL with Distribution Robust Optimization (DRO), resulting in robust models that perform well on all clients (Mohri et al, 2019;Deng et al, 2021); combining FL with techniques that learn domain invariant features (Peng et al, 2019;Wang et al, 2022a;Shen et al, 2021;Sun et al, 2022;Gan et al, 2021) to improve the generalization ability of trained models. All of the above methods aim to train a single robust feature extractor that can generalize well on unseen distributions.…”

“…However, current research in FL primarily focuses on addressing label distribution skew (Li et al, 2018b;Wang et al, 2020b;Karimireddy et al, 2020b), leaving the issues of feature distribution skew and concept shift among clients under-explored. Only a limited number of studies have investigated this issue in specific settings, either training robust feature extractors when train and test datasets have different feature distribution (Peng et al, 2019;Wang et al, 2022a;Shen et al, 2021), or addressing concept shifts under some specified conditions (Jothimurugesan et al, 2022). These attempts are far behind in addressing the challenges of the simultaneous occurrence of feature distribution, label distribution, and concept shifts among clients that naturally emerged in FL (as shown in Figure 1), due to the non-trivial diverse distribution shift challenge: we do not know if there exist distribution shifts among clients, as well as the type of these distribution shifts.…”

Federated Minimax Optimization: Improved Convergence Analyses and Algorithms

“…Some recent works have attempted to achieve this goal for convex-concave Deng et al (2020); Hou et al (2021); Liao et al (2021), for nonconvex-concave Deng et al (2020), and for nonconvex-nonconcave problems Deng & Mahdavi (2021); Reisizadeh et al (2020); Guo et al (2020); Yuan et al (2021). However, in the context of stochastic smooth nonconvex minimax problems, the convergence guarantees of the existing distributed/federated approaches are, to the best of our knowledge, either asymptotic Shen et al (2021) or suboptimal Deng & Mahdavi (2021). In particular, they do not reduce to the existing baseline results for the centralized minimax problems (n = 1).…”