Communication Efficient Federated Learning over Multiple Access Channels

Abstract

In this work, we study the problem of federated learning (FL), where distributed users aim to jointly train a machine learning model with the help of a parameter server (PS). In each iteration of FL, users compute local gradients, followed by transmission of the quantized gradients for subsequent aggregation and model updates at PS. One of the challenges of FL is that of communication overhead due to FL's iterative nature and large model sizes. One recent direction to alleviate communication bottleneck in FL is to let users communicate simultaneously over a multiple access channel (MAC), possibly making better use of the communication resources.In this paper, we consider the problem of FL learning over a MAC. In particular, we focus on the design of digital gradient transmission schemes over a MAC, where gradients at each user are first quantized, and then transmitted over a MAC to be decoded individually at the PS. When designing digital FL schemes over MACs, there are new opportunities to assign different amount of resources (such as rate or bandwidth) to different users based on a) the informativeness of the gradients at each user, and b) the underlying channel conditions. We propose a stochastic gradient quantization scheme, where the quantization parameters are optimized based on the capacity region of the MAC. We show that such channel aware quantization for FL outperforms uniform quantization, particularly when users experience different channel conditions, and when have gradients with varying levels of informativeness.

Enabling Long-Term Cooperation in Cross-Silo Federated Learning: A Repeated Game Perspective

“…, G. Since clients in cross-silo FL are companies or organizations, the data transmission between clients and the central server can be through either the wired networks (e.g., through the high-speed wired connections [41]) or the wireless networks (e.g., using transmission protocols TDMA or OFDMA [37]). We assume that each client has the same communication resource [42], and the parameters of local models have the same size. In this case, clients experience the same communication cost (e.g., transmission delay), which depends on the number of iterations between clients and the central server [40].…”

Timely Communication in Federated Learning

“…In (11), moments of S are not easy to calculate as S is equal to an order statistic of a sum of exponential and shifted exponential random variables. To simplify, we assume that downlink transmissions are instantaneous since, in general, connection speeds are significantly asymmetric such that downlink transmissions are much faster than uplink transmissions [23].…”

“…Some promising applications of FL are image classification and nextword prediction [1], human mobility prediction [2], news recommenders and interactive social networks [3], healthcare applications [4], and so on. Recent works in [5]- [11] study communication-efficient FL frameworks suitable for the limited communication between the PS and the clients considering varying channel conditions, quantization and sparsification, non-i.i.d. client datasets, and coding.…”

Linear Regression over Networks with Communication Guarantees

“…Lazy updates are introduced in (Chen et al, 2018a,b), and combinations of non-periodic updates and quantization is explored in (Reisizadeh et al, 2020). Furthermore, when distributed learning is taking place over a wireless network, there is an interest in allocating the available network resources efficiently among the users holding the data (Gündüz et al, 2019), such as power (Chen et al, 2019) or rates (Chang and Tandon, 2020). The problem of scheduling gradient updates over multiple access channels has also received initial consideration, for example comparing time-based approaches with approaches based on channel conditions , or including gradient information (Amiri et al, 2020;Chen et al, 2020).…”