Semi-Supervised Federated Learning with non-IID Data: Algorithm and System Design

Abstract

Federated Learning (FL) allows edge devices (or clients) to keep data locally while simultaneously training a shared high-quality global model. However, current research is generally based on an assumption that the training data of local clients have ground-truth. Furthermore, FL faces the challenge of statistical heterogeneity, i.e., the distribution of the client's local training data is non-independent identically distributed (non-IID). In this paper, we present a robust semi-supervised FL system design, where the system aims to solve the problem of data availability and non-IID in FL. In particular, this paper focuses on studying the labels-at-server scenario where there is only a limited amount of labeled data on the server and only unlabeled data on the clients. In our system design, we propose a novel method to tackle the problems, which we refer to as Federated Mixing (FedMix). FedMix improves the naive combination of FL and semi-supervised learning methods and designs parameter decomposition strategies for disjointed learning of labeled, unlabeled data, and global models. To alleviate the non-IID problem, we propose a novel aggregation rule based on the frequency of the client's participation in training, namely the FedFreq aggregation algorithm, which can adjust the weight of the corresponding local model according to this frequency. Extensive evaluations conducted on CIFAR-10 dataset show that the performance of our proposed method is significantly better than those of the current baseline. It is worth noting that our system is robust to different non-IID levels of client data.

Knowledge-Enhanced Semi-Supervised Federated Learning for Aggregating Heterogeneous Lightweight Clients in IoT

“…• RQ3: How does each module take effects in the framework to achieve personalized efficiency federated learning under the semi-supervised setting? SSFL [39], FedMix [38], FedSEAL [1], and SemiFL [4] as baselines.…”

“…As there is limited work [14] on federated learning in natural language processing, we choose the identical baselines with image tasks. However, according to the intuition and typical design for image tasks, SSFL [39] and FedMix [38] are not fit for text tasks. SSFL is based on replacing batch normalization with group normalization.…”

“…In [8], a novel consistency loss is proposed to maximize the agreement between different local models, and disjoint learning is applied to extract reliable knowledge from the labeled and unlabeled data. [38,39] design the group-based and weighted-based aggregation strategy to decrease the negative effect of data heterogeneity. In [1], the authors add self-ensemble learning and complementary negative learning to federated learning.…”

“…Recently, a few SemiFL approaches have been proposed, such as FedMatch [8], SSFL [39], FedMix [38], FedSEAL [1], SemiFL [4], and FedTriNet [2]. These models primarily focus on developing a good global model and serving one global model to all clients.…”