Multi-Job Intelligent Scheduling With Cross-Device Federated Learning

Abstract

Recent years have witnessed a large amount of decentralized data in various (edge) devices of end-users, while the decentralized data aggregation remains complicated for machine learning jobs because of regulations and laws. As a practical approach to handling decentralized data, Federated Learning (FL) enables collaborative global machine learning model training without sharing sensitive raw data. The servers schedule devices to jobs within the training process of FL. In contrast, device scheduling with multiple jobs in FL remains a critical and open problem. In this paper, we propose a novel multi-job FL framework, which enables the training process of multiple jobs in parallel. The multi-job FL framework is composed of a system model and a scheduling method. The system model enables a parallel training process of multiple jobs, with a cost model based on the data fairness and the training time of diverse devices during the parallel training process. We propose a novel intelligent scheduling approach based on multiple scheduling methods, including an original reinforcement learning-based scheduling method and an original Bayesian optimization-based scheduling method, which corresponds to a small cost while scheduling devices to multiple jobs. We conduct extensive experimentation with diverse jobs and datasets. The experimental results reveal that our proposed approaches significantly outperform baseline approaches in terms of training time (up to 12.73 times faster) and accuracy (up to 46.4% higher).

Efficient asynchronous federated learning with sparsification and quantization

“…In order to address the communication bottleneck with FL, diverse techniques 32–41 can be exploited, for example, sparsification, quantization, and pruning. Model sparsification transfer has been widely exploited in distributed machine learning 14,15,42–44 .…”

Distributed and Deep Vertical Federated Learning with Big Data

“…Their approach is classified into horizontal FL years later, because of the wide range of participants and similar data features, e.g., health care with the attention mechanism on graphs 28 , 5G-empowered drone networks with reinforcement learning for smart grid or smart cities 29 , frequent itemset mining 30 , distributed medical data 31 and education data 32 , purchase behaviour with an attention mechanism 33 , and Internet of Things (IoT) devices 34,31 . In addition, the horizontal FL can be carried out in the Cloud for extreme gradient boosting 35 or deep learning models with the combination of synchronous 36,37,38,39,40 and asynchronous mechanisms 41,42 . Later, the first FL on vertically partitioned data was proposed to train a LR model 9 , which exploits HE to protect data security.…”

Distributed and deep vertical federated learning with big data

“…Their approach is classified into horizontal FL years later, because of the wide range of participants and similar data features, for example, health care with the attention mechanism on graphs, 28 5G-empowered drone networks with reinforcement learning for smart grid or smart cities, 29 frequent itemset mining, 30 distributed medical data 31 and education data, 32 purchase behavior with an attention mechanism, 33 and Internet of Things (IoT) devices. 31,34 In addition, the horizontal FL can be carried out in the cloud for extreme gradient boosting 35 or deep learning models with the combination of synchronous [36][37][38][39][40] and asynchronous mechanisms. 41,42 Later, the first FL on vertically partitioned data was proposed to train a LR model, 9 which exploits HE to protect data security.…”