FedSiam: Towards Adaptive Federated Semi-Supervised Learning

Long, Zewei; Che, Liwei; Wang, Yaqing; Ye, Muchao; Jian, Luo; Wu, Jinze; Xiao, Houping; Ma, Fenglong

doi:10.48550/arxiv.2012.03292

Cited by 11 publications

(16 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Label deficiency in FL. There are a few related works to tackle label deficiency in FL , Long et al, 2020, Itahara et al, 2020, Jeong et al, 2020, Liang et al, 2021, Zhang et al, 2020b. Compared to these works, our proposed SSFL does not use any labels during training.…”

Section: Related Workmentioning

confidence: 99%

“…These algorithms all depend on the strong assumption that the data at the edge has sufficient labels. To address the label deficiency issue in FL, recent works , Long et al, 2020, Itahara et al, 2020, Jeong et al, 2020, Liang et al, 2021, Zhang et al, 2020a assume that the server or client has a fraction of labeled data and use semi-supervised methods such as consistency loss [Miyato et al, 2018] or pseudo labeling [Lee, 2013] to train a global model. A more realistic but challenging setting is fully unsupervised training.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision

He,

Yang,

Mushtaq

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated Learning (FL) is transforming the ML training ecosystem from a centralized over-thecloud setting to distributed training over edge devices in order to strengthen data privacy, reduce data migration costs, and break regulatory restrictions. An essential, but rarely studied, challenge in FL is label deficiency at the edge. This problem is even more pronounced in FL, compared to centralized training, due to the fact that FL users are often reluctant to label their private data and edge devices do not provide an ideal interface to assist with annotation. Addressing label deficiency is also further complicated in FL, due to the heterogeneous nature of the data at edge devices and the need for developing personalized models for each user. We propose a self-supervised and personalized federated learning framework, named (SSFL), and a series of algorithms under this framework which work towards addressing these challenges. First, under the SSFL framework, we analyze the compatibility of various centralized self-supervised learning methods in FL setting and demonstrate that SimSiam networks performs the best with the standard FedAvg algorithm. Moreover, to address the data heterogeneity at the edge devices in this framework, we have innovated a series of algorithms that broaden existing supervised personalization algorithms into the setting of self-supervised learning including perFedAvg, Ditto, and local fine-tuning, among others. We further propose a novel personalized federated self-supervised learning algorithm, Per-SSFL, which balances personalization and consensus by carefully regulating the distance between the local and global representations of data. To provide a comprehensive comparative analysis of all proposed algorithms, we also develop a distributed training system and related evaluation protocol for SSFL. Using this training system, we conduct experiments on a synthetic non-I.I.D. dataset based on CIFAR-10, and an intrinsically non-I.I.D. dataset GLD-23K. Our findings show that the gap of evaluation accuracy between supervised learning and unsupervised learning in FL is both small and reasonable. The performance comparison indicates that representation regularization-based personalization method is able to outperform other variants. Ablation studies on SSFL are also conducted to understand the role of batch size, non-I.I.D.ness, and the evaluation protocol.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision

He,

Yang,

Mushtaq

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Semi-supervised federated learning attempts to use semisupervised learning techniques [19]- [23] to further improve the performance of the FL model in scenarios where there is unlabeled data on the client side [11]. For example, Long et al in [10] proposed a semi-supervised federated learning (SSFL) system, FedSemi, which unifies the consistency-based semisupervised learning model [24], dual model [15], and average teacher model [25] to achieve SSFL. The DS-FL system [26] was proposed to solve the communication overhead problem in SSFL.…”

Section: A Semi-supervised Federated Learningmentioning

confidence: 99%

“…Reference [27] proposes a method to study the distribution of non-IID data, which introduces a probability distance metric to evaluate the difference in client data distribution in SSFL. Different from the literature [9], [10], [26], in this paper, we focus on labels-at-server scenario and also solve the problem of data availability and data heterogeneity in SSFL.…”

Section: A Semi-supervised Federated Learningmentioning

confidence: 99%

“…In fact, consistency regularization technique is widely used in semi-supervised learning which keeps the same output that the same data inject two different noises [15], [16]. Furthermore, pseudo-label methods are important for SSFL, where they mainly utilize pseudo-labels whose predicted value is higher than the confidence threshold to achieve highprecision SSFL [9], [10], [17]. However, it is worth noting that there still remain gaps when deploying SSFL in practice.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Semi-Supervised Federated Learning with non-IID Data: Algorithm and System Design

Zhang¹,

Ma²,

Nie³

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated Learning (FL) allows edge devices (or clients) to keep data locally while simultaneously training a shared high-quality global model. However, current research is generally based on an assumption that the training data of local clients have ground-truth. Furthermore, FL faces the challenge of statistical heterogeneity, i.e., the distribution of the client's local training data is non-independent identically distributed (non-IID). In this paper, we present a robust semi-supervised FL system design, where the system aims to solve the problem of data availability and non-IID in FL. In particular, this paper focuses on studying the labels-at-server scenario where there is only a limited amount of labeled data on the server and only unlabeled data on the clients. In our system design, we propose a novel method to tackle the problems, which we refer to as Federated Mixing (FedMix). FedMix improves the naive combination of FL and semi-supervised learning methods and designs parameter decomposition strategies for disjointed learning of labeled, unlabeled data, and global models. To alleviate the non-IID problem, we propose a novel aggregation rule based on the frequency of the client's participation in training, namely the FedFreq aggregation algorithm, which can adjust the weight of the corresponding local model according to this frequency. Extensive evaluations conducted on CIFAR-10 dataset show that the performance of our proposed method is significantly better than those of the current baseline. It is worth noting that our system is robust to different non-IID levels of client data.

show abstract

Migrating federated learning to centralized learning with the leverage of unlabeled data

et al. 2023

View full text Add to dashboard Cite

Federated learning carries out cooperative training without local data sharing, the obtained global model performs generally better than independent local models. Benefiting from the free data sharing, federated learning preserves the privacy of local users. However, the performance of the global model might be degraded if diverse clients hold non-IID training data. This is because the different distributions of local data lead to weight divergence of local models. In this paper, we introduce a novel teacher-student framework to alleviate the negative impact of non-IID data. On the one hand, we maintain the advantage of the federated learning on the privacy-preserving, and on the other hand, we take the advantage of the centralized learning on the accuracy. We use unlabeled data and global models as teachers to generate a pseudo-labeled dataset, which can significantly improve the performance of the global model. At the same time, the global model as a teacher provides more accurate pseudo labels. In addition, we perform a model rollback to mitigate the impact of latent noise labels and data imbalance in the pseudo-labeled dataset. Extensive experiments have verified that our teacher ensemble performs a more robust training. The empirical study verifies that the reliance on the centralized pseudolabeled data enables the global model almost immune to non-IID data.

show abstract

FedSiam: Towards Adaptive Federated Semi-Supervised Learning

Cited by 11 publications

References 22 publications

SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision

SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision

Semi-Supervised Federated Learning with non-IID Data: Algorithm and System Design

Migrating federated learning to centralized learning with the leverage of unlabeled data

Contact Info

Product

Resources

About