Astraea: Self-Balancing Federated Learning for Improving Classification Accuracy of Mobile Deep Learning Applications

Abstract

Federated learning (FL) is a distributed deep learning method which enables multiple participants, such as mobile phones and IoT devices, to contribute a neural network model while their private training data remains in local devices. This distributed approach is promising in the edge computing system where have a large corpus of decentralized data and require high privacy. However, unlike the common training dataset, the data distribution of the edge computing system is imbalanced which will introduce biases in the model training and cause a decrease in accuracy of federated learning applications. In this paper, we demonstrate that the imbalanced distributed training data will cause accuracy degradation in FL. To counter this problem, we build a self-balancing federated learning framework call Astraea, which alleviates the imbalances by 1) Global data distribution based data augmentation, and 2) Mediator based multi-client rescheduling. The proposed framework relieves global imbalance by runtime data augmentation, and for averaging the local imbalance, it creates the mediator to reschedule the training of clients based on KullbackLeibler divergence (KLD) of their data distribution. Compared with FedAvg, the state-of-the-art FL algorithm, Astraea shows +5.59% and +5.89% improvement of top-1 accuracy on the imbalanced EMNIST and imbalanced CINIC-10 datasets, respectively. Meanwhile, the communication traffic of Astraea can be 92% lower than that of FedAvg.• We first find out that the global imbalanced training data will degrade the accuracy of CNN models trained by FL.• We propose a self-balancing federated learning framework, Astraea, along with two strategies to prevent the

Client Selection for Federated Learning With Non-IID Data in Mobile Edge Computing

“…We adopt a convolutional neural network (CNN), which is a class of deep neural networks and commonly applied in the literature [8], [12], [13], [23], [24]. Specifically, we set four 3 × 3 convolution layers (32, 32, 64, 64 channels, each of which was activated by ReLU and batch normalized, and every two of which were followed by 2 × 2 max pooling), two fully connected layers (384 and 192 units with ReLU activation), and a final output layer with 10 units for training both CIFAR-10 and Fashion MNIST datasets.…”

“…Besides local imbalance, global imbalance may also degrade the training accuracy of FL [24]. We now evaluate the impact of such global imbalance on the performance of the proposed CSFedAvg.…”

“…However, as the training data on a given client typically relies on the usage of the mobile device by a particular user, the client's local dataset may not be representative of the population distribution, i.e., non-independent and identically distribution (non-IID). Such an imbalanced data distribution incurred by non-IID data will bring the biases in the model training, and may cause an accuracy degradation of FL [8], [17]- [24]. To explore the influence of non-IID data on FL, [8] proposes a communication-efficient FL algorithm, named FedAvg, and evaluates its performance on non-IID MNIST dataset.…”

“…However, the random selection of clients to upload their data may not form the IID dataset, due to the heterogeneous of clients on their data. The work in [24] assumes that the clients are willing to send their data distribution/label information to the server. Based on this, [24] uses mediators to reschedule the training of clients, so as to improve the accuracy of FedAvg on imbalanced data.…”

“…The work in [24] assumes that the clients are willing to send their data distribution/label information to the server. Based on this, [24] uses mediators to reschedule the training of clients, so as to improve the accuracy of FedAvg on imbalanced data. However, clients might still be reluctant to share their data distribution or class with others, for the privacy reasons.…”