Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Sampling

Abstract

Federated learning (FL) algorithms usually sample a fraction of clients in each round (partial participation) when the number of participants is large and the server's communication bandwidth is limited. Recent works on the convergence analysis of FL have focused on unbiased client sampling, e.g., sampling uniformly at random, which suffers from slow wall-clock time for convergence due to high degrees of system heterogeneity and statistical heterogeneity. This paper aims to design an adaptive client sampling algorithm that tackles both system and statistical heterogeneity to minimize the wall-clock convergence time. We obtain a new tractable convergence bound for FL algorithms with arbitrary client sampling probabilities. Based on the bound, we analytically establish the relationship between the total learning time and sampling probabilities, which results in a non-convex optimization problem for training time minimization. We design an efficient algorithm for learning the unknown parameters in the convergence bound and develop a low-complexity algorithm to approximately solve the non-convex problem. Experimental results from both hardware prototype and simulation demonstrate that our proposed sampling scheme significantly reduces the convergence time compared to several baseline sampling schemes. Notably, our scheme in hardware prototype spends 73% less time than the uniform sampling baseline for reaching the same target loss.3 We use wall-clock time to distinguish from the number of training rounds.

Federated Learning Design and FunctionalModels: Survey

“…Additionally, the Power-of-Choice [39] method has been used for managing biased client selection and has received better convergence rates compared to the random client selection method. Better performance in terms of faster convergence and more consistent management is made possible using methods like arbitrary client selection [44] and adaptive client selection [45] by assessing and updating probabilities pertaining to the client relationships. Using these types of client selection can help manage unknown parameters used for solving complex problems.…”

Aprendizado Federado com Agrupamento Hierárquico de Clientes para Aumento da Acurácia

“…Luo et al [Luo et al 2021] e Lai et al [Lai et al 2021] propõem esquemas de selec ¸ão de clientes que buscam otimizar a velocidade de convergência do modelo em ambientes de aprendizado federado. Os autores argumentam que a selec ¸ão baseada apenas na representatividade dos dados diminui o número total de épocas para convergência do modelo.…”