Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies

Cho, Yae Jee; Wang, Yunlong; Joshi, Gauri

doi:10.48550/arxiv.2010.01243

Cited by 63 publications

(120 citation statements)

References 17 publications

Supporting

Mentioning

118

Contrasting

Order By: Relevance

“…Among these is [17], but only the convergence of simple linear regression loss is considered. In [18], the authors analyze the convergence of strongly convex loss functions, but unfortunately their bound introduces a nonvanishing term and thus their strategy is not guaranteed to converge to a stationary point of the loss function. Both [19] and [20] consider convergence, but only for strongly convex loss functions.…”

Section: R Wmentioning

confidence: 99%

“…The user-defined parameter V traditionally controls the trade-off between the average queue backlog and the gap from optimality, but since we do not have physical queues in our problem, the trade-off does not exist in the same way. Instead, V controls the speed of convergence in addition to the optimality gap in (18).…”

Section: Algorithm 2: Stochastic Client Samplingmentioning

confidence: 99%

See 1 more Smart Citation

Communication-Efficient Device Scheduling for Federated Learning Using Stochastic Optimization

Perazzone¹,

Wang²,

Ji³

et al. 2022

Preprint

View full text Add to dashboard Cite

Federated learning (FL) is a useful tool in distributed machine learning that utilizes users' local datasets in a privacy-preserving manner. When deploying FL in a constrained wireless environment; however, training models in a time-efficient manner can be a challenging task due to intermittent connectivity of devices, heterogeneous connection quality, and non-i.i.d. data. In this paper, we provide a novel convergence analysis of nonconvex loss functions using FL on both i.i.d. and non-i.i.d. datasets with arbitrary device selection probabilities for each round. Then, using the derived convergence bound, we use stochastic optimization to develop a new client selection and power allocation algorithm that minimizes a function of the convergence bound and the average communication time under a transmit power constraint. We find an analytical solution to the minimization problem. One key feature of the algorithm is that knowledge of the channel statistics is not required and only the instantaneous channel state information needs to be known. Using the FEMNIST and CIFAR-10 datasets, we show through simulations that the communication time can be significantly decreased using our algorithm, compared to uniformly random participation.

show abstract

Section: R Wmentioning

confidence: 99%

Section: Algorithm 2: Stochastic Client Samplingmentioning

confidence: 99%

Communication-Efficient Device Scheduling for Federated Learning Using Stochastic Optimization

Perazzone¹,

Wang²,

Ji³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Specifically, clients with "important" data would have higher probabilities to be sampled in each round. For example, existing works use clients' local gradient information (e.g., [25]- [27]) or local losses (e.g., [28]) to measure the importance of clients' data. However, these schemes did not consider the speed of error convergence with respect to wall-clock time, especially the straggling effect due to heterogeneous transmission delays.…”

Section: Related Workmentioning

confidence: 99%

“…Assumptions 1-3 are common in many existing studies of convex FL problems, such as 2 -norm regularized linear regression, logistic regression (e.g., [7], [18], [19], [25], [28], [42]). Nevertheless, the experimental results to be presented in Section VI show that our approach also works well for non-convex loss functions.…”

Section: A Machine Learning Model Assumptionsmentioning

confidence: 99%

“…One effective way of speeding up the convergence with respect to the number of training rounds is to choose clients according to some sampling distribution where "important" clients have high probabilities [21]- [24]. For example, recent works adopted importance sampling approaches based on clients' statistical property [25]- [28]. However, their sampling schemes did not account for the heterogeneous physical time in each round, especially under straggling circumstances.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Sampling

Luo¹,

Xiao²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated learning (FL) algorithms usually sample a fraction of clients in each round (partial participation) when the number of participants is large and the server's communication bandwidth is limited. Recent works on the convergence analysis of FL have focused on unbiased client sampling, e.g., sampling uniformly at random, which suffers from slow wall-clock time for convergence due to high degrees of system heterogeneity and statistical heterogeneity. This paper aims to design an adaptive client sampling algorithm that tackles both system and statistical heterogeneity to minimize the wall-clock convergence time. We obtain a new tractable convergence bound for FL algorithms with arbitrary client sampling probabilities. Based on the bound, we analytically establish the relationship between the total learning time and sampling probabilities, which results in a non-convex optimization problem for training time minimization. We design an efficient algorithm for learning the unknown parameters in the convergence bound and develop a low-complexity algorithm to approximately solve the non-convex problem. Experimental results from both hardware prototype and simulation demonstrate that our proposed sampling scheme significantly reduces the convergence time compared to several baseline sampling schemes. Notably, our scheme in hardware prototype spends 73% less time than the uniform sampling baseline for reaching the same target loss.3 We use wall-clock time to distinguish from the number of training rounds.

show abstract

FLight: A lightweight federated learning framework in edge and fog computing

Zhu,

Goudarzi,

Buyya

2023

Softw Pract Exp

View full text Add to dashboard Cite

The number of Internet of Things (IoT) applications, especially latency‐sensitive ones, have been significantly increased. So, cloud computing, as one of the main enablers of the IoT that offers centralized services, cannot solely satisfy the requirements of IoT applications. Edge/fog computing, as a distributed computing paradigm, processes, and stores IoT data at the edge of the network, offering low latency, reduced network traffic, and higher bandwidth. The edge/fog resources are often less powerful compared to cloud, and IoT data is dispersed among many geo‐distributed servers. Hence, Federated Learning (FL), which is a machine learning approach that enables multiple distributed servers to collaborate on building models without exchanging the raw data, is well‐suited to edge/fog computing environments, where data privacy is of paramount importance. Besides, to manage different FL tasks on edge/fog computing environments, a lightweight resource management framework is required to manage different incoming FL tasks while does not incur significant overhead on the system. Accordingly, in this article, we propose a lightweight FL framework, called FLight, to be deployed on a diverse range of devices, ranging from resource‐limited edge/fog devices to powerful cloud servers. FLight is implemented based on the FogBus2 framework, which is a containerized distributed resource management framework. Moreover, FLight integrates both synchronous and asynchronous models of FL. Besides, we propose a lightweight heuristic‐based worker selection algorithm to select a suitable set of available workers to participate in the training step to obtain higher training time efficiency. The obtained results demonstrate the efficiency of the FLight. The worker selection technique reduces the training time of reaching 80% accuracy by 34% compared to sequential training, while asynchronous one helps to improve synchronous FL training time by 64%.

show abstract

Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies

Cited by 63 publications

References 17 publications

Communication-Efficient Device Scheduling for Federated Learning Using Stochastic Optimization

Communication-Efficient Device Scheduling for Federated Learning Using Stochastic Optimization

Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Sampling

FLight: A lightweight federated learning framework in edge and fog computing

Contact Info

Product

Resources

About