Semi-HFL: semi-supervised federated learning for heterogeneous devices

Abstract

In the vanilla federated learning (FL) framework, the central server distributes a globally unified model to each client and uses labeled samples for training. However, in most cases, clients are equipped with different devices and are exposed to a variety of situations. There are great differences between clients in storage, computing, communication, and other resources, which makes unified deep models used in traditional FL cannot fit clients’ personalized resource conditions. Furthermore, a great deal of labeled data is needed in traditional FL, whereas data labeling requires a great investment of time and resources, which is hard to do for individual clients. As a result, clients only have a vast amount of unlabeled data, which goes against the federated learning needs. To address the aforementioned two issues, we propose Semi-HFL, a semi-supervised federated learning approach for heterogeneous devices, which divides a deep model into a series of small submodels by inserting early exit branches to meet the resource requirements of different devices. Furthermore, considering the availability of labeled data, Semi-HFL introduces semi-supervised techniques for training in the above heterogeneous learning process. Specifically, two training phases are included in the semi-supervised learning process, unsupervised learning on clients and supervised learning on the server, which makes full use of clients’ unlabeled data. Through image classification, text classification, next-word prediction, and multi-task FL experiments based on five kinds of datasets, it is verified that compared with the traditional homogeneous learning method, Semi-HFL not only achieves higher accuracies but also significantly reduces the global resource overhead.

Semi-supervised Graph Instance Transformer for Mental Health Inference

“…To tackle the problem of lack of labels in clients, semi-supervised learning is integrated to the FL framework to leverage the large amount of unlabeled data. Zhong et al proposed a semi-supervised federated learning method for heterogeneous devices, which adopts pseudo-labeling method as its semi-supervised learning strategy [32]. Specifically, it assumes that the central server has a small amount of labeled data to train an initial global model, which is then used to obtain pseudo-labels for local unlabeled samples that meet certain entropy conditions (i.e., below a selected entropy threshold).…”