Communication-Efficient Federated Learning via Optimal Client Sampling

Abstract

Federated learning is a private and efficient framework for learning models in settings where data is distributed across many clients. Due to interactive nature of the training process, frequent communication of large amounts of information is required between the clients and the central server which aggregates local models. We propose a novel, simple and efficient way of updating the central model in communication-constrained settings by determining the optimal client sampling policy. In particular, modeling the progression of clients' weights by an Ornstein-Uhlenbeck process allows us to derive the optimal sampling strategy for selecting a subset of clients with significant weight updates. The central server then collects local models from only the selected clients and subsequently aggregates them. We propose four client sampling strategies and test them on two federated learning benchmark tests, namely, a classification task on EMNIST and a realistic language modeling task using the Stackoverflow dataset. The results show that the proposed framework provides significant reduction in communication while maintaining competitive or achieving superior performance compared to baseline. Our methods introduce a new line of communication strategies orthogonal to the existing user-local methods such as quantization or sparsification, thus complementing rather than aiming to replace them.

Communication-Efficient Device Scheduling for Federated Learning Using Stochastic Optimization

“…datasets as indicated in their results. Some other empirical studies with similar approaches include [15], [16], but also do not consider or derive convergence bounds for their selection strategies. Some later work began to include analysis of the convergence of FL with device selection.…”

“…Since the objective is an independent sum over n, we can perform the minimization separately for each device n. Algorithm 2 details to process in determining the optimal P n (t) and q t n in each round. We now present Theorem 2 which gives an analytical solution to (15) that can be computed distributively by the devices.…”

“…Theorem 2. The solution to (15) is given by Algorithm 2 where the optimal values for each n is given by either the endpoints, i.e., P opt n (t) = P max , q t n = 1 or by…”

“…To find the roots, we compute the gradient of the objective function for each n in (15) ∇f (q t n , P n (t)) =   …”

Client Selection in Federated Learning based on Gradients Importance

“…They propose a federated averaging scheme where the aggregation is weighted by probabilities of devices being inactive at a given communication round. Another selection approach is proposed in [9]. The approach suggests that clients with the most significant local updates are selected.…”