A Sequential Gradient-Based Multiple Access for Distributed Learning over Fading Channels

Sery, Tomer; Cohen, Kobi

doi:10.1109/allerton.2019.8919883

Cited by 10 publications

(5 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Summary Training with Noisy Gradients [137] Proposal of gradient-based multiple-access scheme that does not cancel the fading effect but operates directly with noisy gradients. [138] Convergence rate analysis for gradientbased multiple-access.…”

Section: Topicmentioning

confidence: 99%

See 1 more Smart Citation

Wireless for Machine Learning: A Survey

Hellström¹,

Silva²,

Amiri³

et al. 2022

View full text Add to dashboard Cite

As data generation increasingly takes place on devices without a wired connection, Machine Learning (ML) related traffic will be ubiquitous in wireless networks. Many studies have shown that traditional wireless protocols are highly inefficient or unsustainable to support ML, which creates the need for new wireless communication methods. In this monograph, we give a comprehensive review of the state-of-the-art wireless methods that are specifically designed to support ML services over distributed datasets. Currently, there are two clear themes within the literature, analog over-the-air computation and digital radio resource management optimized for ML. This survey gives an introduction to these methods, reviews the most important works, highlights open problems, and discusses application scenarios.

show abstract

Section: Topicmentioning

confidence: 99%

“…Unlike all papers we have surveyed so far, which used channel inversion to combat fading, see Eq. (3.2), the authors of [137] suggest just transmitting without doing any precoding. Such a scheme has the advantage of not requiring a channel estimate, and a generally simpler implementation.…”

Section: Training With Noisy Gradientsmentioning

confidence: 99%

Wireless for Machine Learning: A Survey

Hellström¹,

Silva²,

Amiri³

et al. 2022

View full text Add to dashboard Cite

show abstract

“…To deal with FL's iterative property and large model size, one popular direction is to study FL over MAC (FL-MAC). Communication inefficiency in FL can be alleviated through communicating over MAC so that the communication resources can be used more effectively [17,18].…”

Section: Multiple Access Channelsmentioning

confidence: 99%

“…FL has been considered in different communication scenarios [8][9][10][11][12][13][14][15]. Some people study joint model devices through wireless channels [9,11,[13][14][15][16], some people consider FL on multiple access channels (MAC) [8,10,17,18], which is denoted as FL-MAC.…”

Section: Introductionmentioning

confidence: 99%

FL-MAC-RDP: Federated Learning over Multiple Access Channels with Rényi Differential Privacy

Ahmed

et al. 2021

Int J Theor Phys

View full text Add to dashboard Cite

Federated Learning (FL) is a promising paradigm, where the local users collaboratively learn models by repeatedly sharing information while the data is kept distributing on these users. FL has been considered in multiple access channels (FL-MAC), which is a hot issue. Even though FL-MAC has many advantages, it is still possible to leak privacy to a third party during the whole training process. To avoid privacy leakage, we propose to add Rényi differential privacy (RDP) into FL-MAC. At the same time, to maximize the convergent rate of users under the constraints of transmission rate and privacy, the quantization stochastic gradient descent (QSGD) is performed by users. We also illustrate our results on MNIST, and the illustration demonstrate that our scheme can improve the model accuracy with a little loss of communication efficiency.

show abstract

“…The superposition nature of wireless channels allows gradients to be aggregated "over-the-air" and allows for much more efficient training. Several recent works include [11][12][13][14][15][16][17][18][19][20][21][22][23]. The approaches can be broadly categorized into digital or analog schemes depending on how the gradients are transmitted over the channel.…”

Section: Introductionmentioning

confidence: 99%

Communication Efficient Federated Learning over Multiple Access Channels

Chang,

Tandon

2020

Preprint

View full text Add to dashboard Cite

In this work, we study the problem of federated learning (FL), where distributed users aim to jointly train a machine learning model with the help of a parameter server (PS). In each iteration of FL, users compute local gradients, followed by transmission of the quantized gradients for subsequent aggregation and model updates at PS. One of the challenges of FL is that of communication overhead due to FL's iterative nature and large model sizes. One recent direction to alleviate communication bottleneck in FL is to let users communicate simultaneously over a multiple access channel (MAC), possibly making better use of the communication resources.In this paper, we consider the problem of FL learning over a MAC. In particular, we focus on the design of digital gradient transmission schemes over a MAC, where gradients at each user are first quantized, and then transmitted over a MAC to be decoded individually at the PS. When designing digital FL schemes over MACs, there are new opportunities to assign different amount of resources (such as rate or bandwidth) to different users based on a) the informativeness of the gradients at each user, and b) the underlying channel conditions. We propose a stochastic gradient quantization scheme, where the quantization parameters are optimized based on the capacity region of the MAC. We show that such channel aware quantization for FL outperforms uniform quantization, particularly when users experience different channel conditions, and when have gradients with varying levels of informativeness.

show abstract

A Sequential Gradient-Based Multiple Access for Distributed Learning over Fading Channels

Cited by 10 publications

References 32 publications

Wireless for Machine Learning: A Survey

Wireless for Machine Learning: A Survey

FL-MAC-RDP: Federated Learning over Multiple Access Channels with Rényi Differential Privacy

Communication Efficient Federated Learning over Multiple Access Channels

Contact Info

Product

Resources

About