Gradient Scheduling with Global Momentum for Non-IID Data Distributed Asynchronous Training

Abstract

Distributed asynchronous offline training has received widespread attention in recent years because of its high performance on largescale data and complex models. As data are distributed from cloudcentric to edge nodes, a big challenge for distributed machine learning systems is how to handle native and natural non-independent and identically distributed (non-IID) data for training. Previous asynchronous training methods do not have a satisfying performance on non-IID data because it would result in that the training process fluctuates greatly which leads to an abnormal convergence. We propose a gradient scheduling algorithm with partly averaged gradients and global momentum (GSGM) for non-IID data distributed asynchronous training. Our key idea is to apply global momentum and local average to the biased gradient after scheduling, in order to make the training process steady. Experimental results show that for non-IID data training under the same experimental conditions, GSGM on popular optimization algorithms can achieve a 20% increase in training stability with a slight improvement in accuracy on Fashion-Mnist and CIFAR-10 datasets. Meanwhile, when expanding distributed scale on CIFAR-100 dataset that results in sparse data distribution, GSGM can perform a 37% improvement on training stability. Moreover, only GSGM can converge well when the number of computing nodes grows to 30, compared to the state-of-the-art distributed asynchronous algorithms. At the same time, GSGM is robust to different degrees of non-IID data.

Aggregation Delayed Federated Learning

“…On the server side, Li et al [15] applied momentum uniformly to the gradients of all clients to stabilize the training process under a non-IID scenario. However, collecting gradients from clients might require more frequent communications than collecting models from clients.…”

“…To our best knowledge, there is no gold standard for evaluating federated algorithms. Generally, there are 3 ways to split the data into training and test sets: splitting all data globally [7,15,26], splitting each client's local data [1,4,24] and splitting clients into training/test groups [5,8,23]. In this work, we adopt the last strategy by assuming no local data can be collected by the server and the server can not manipulate the client's local data.…”

“…Our work focuses on mitigating the impact of non-IID client data distributions. Many existing works [15,16,26,29,31] adopt the FedAvg [20] framework and applied various strategies to handle non-IID data. In this work, we propose a new framework of federated learning with delayed aggregations.…”

Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data

“…Though K-async FL can mitigate the stragglers effect and save total training time [23], there are still two obstacles to be overcome in practice. On the one hand, non-IID datasets generated at different FL clients can impact the model utility [30], [31]. On the other hand, stale gradients may harm the model utility, or even diverge the training process [32], [33].…”

“…The solutions to these two problems have been studied separately. For non-IID data, the essence of the existing solutions, e.g., momentum [31], [34] and variance reduction [35], is to fully utilize all available information for estimating the global data distribution. Hence, gradients from as many clients as possible need to be aggregated to make the aggregated gradients represent the overall data comprehensively.…”