Fast-Convergent Federated Learning

Abstract

Federated learning has emerged recently as a promising solution for distributing machine learning tasks through modern networks of mobile devices. Recent studies have obtained lower bounds on the expected decrease in model loss that is achieved through each round of federated learning. However, convergence generally requires a large number of communication rounds, which induces delay in model training and is costly in terms of network resources. In this paper, we propose a fast-convergent federated learning algorithm, called FOLB, which obtains significant improvements in convergence speed through intelligent sampling of devices in each round of model training. We first theoretically characterize a lower bound on improvement that can be obtained in each round if devices are selected according to the expected improvement their local models will provide to the current global model. Then, we show that FOLB obtains this bound through uniform sampling by weighting device updates according to their gradient information. FOLB is able to handle both communication and computation heterogeneity of devices by adapting the aggregations according to estimates of device's capabilities of contributing to the updates. We evaluate FOLB in comparison with existing federated learning algorithms and experimentally show its improvement in training loss and test accuracy across various machine learning tasks and datasets.

Fast-Convergent Federated Learning with Adaptive Weighting

“…The local gradient, which is correlated with minimizing the local objective, may not align with the direction of approaching the optimal of the global objective. The correlation ∇ (w( )), ∇ (w( )) ∇ (w( )) ∇ (w( )) between the local gradient and the global gradient is a metric to measure their alignment 2 Similar assumption has made in FL context, for example in [13], [14], [17]. In [13], [17], the dissimilarity across local gradients is imposed by an upper bound to capture the impact of data heterogeneity on FL convergence, and an analogous definition named gradient divergence is also presented in [14].…”

“…Generally, the FL algorithm adopts synchronous aggregation and selects a subset of nodes randomly to participate in each round randomly to avoid long-tailed waiting time due to the network uncertainty and straggler. To boost convergence and reduce the communication rounds, tuning the number of local updates [8], [13]- [15], and selecting appropriate nodes for FL training [12], [16], [17] are the usually adopted approaches.…”

“…Regarding the node selection, Nishio et al [16] proposed FedCS algorithm to do node selection intentionally rather than randomly, based on the resource conditions of local nodes. Authors in [17] utilized gradient information to do node selection. The nodes whose inner product between its gradient vector and the global gradient vector is negative will be excluded from FL training.…”

“…The proposed adaptive weighting strategy can effectively speed up the convergence of FL in the presence of non-IID data. Different from [17], [18], our method does not impose additional communication and computation burden to local nodes. Besides, our adaptive weighting strategy is done in each communication round, which is orthogonal with the methods proposed in [19], [20].…”

“…For theoretical analysis of federated learning algorithms, we employ the following typical assumptions in our analysis (see e.g., [11], [13], [14], [17]). Assumption 1.…”