Analog Compression and Communication for Federated Learning over Wireless MAC

Abdi, Afshin; Saidutta, Yashas Malur; Fekri, Faramarz

doi:10.1109/spawc48557.2020.9154309

Cited by 20 publications

(17 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By contrast to existing over-the-air gradient-based learning methods that compute the gradient directly with respect to the last update, in AGMA, each node computes a momentumbased gradient that uses the last two updated models. AGMA is advantageous in terms of practical implementations, since it does not use power control or beamforming to cancel the channel gain effect as in [3]- [5], [7], [9], [11]- [13]. It should be noted that schemes that correct the channel gains (for instance, by dividing the gradient signal at the transmitters by the channel gain to avoid distortion at the receiver) might still suffer from channel estimation errors.…”

Section: B Main Resultsmentioning

confidence: 99%

“…Channel communication characteristics have been further studied in [9]. In our previous work [8], we have developed and analyzed gradient-based learning without using power control or beamforming to cancel the fading effect.…”

Section: A Related Workmentioning

confidence: 99%

“…A key difference in AGMA is that by contrast to existing methods that compute the gradient with respect to the last update directly, in AGMA, each node computes a momentum-based gradient. Furthermore, AGMA does not use power control or beamforming to cancel the channel gain effect as in [3]- [5], [7], [9], [11]- [13]. The PS updates the model based on the noisy distorted momentum-based gradient directly.…”

Section: The Proposed Accelerated Gradient-descent Multiple Access (A...mentioning

confidence: 99%

See 2 more Smart Citations

Accelerated Gradient Descent Learning over Multiple Access Fading Channels

Paul¹,

Friedman²,

Cohen³

2021

Preprint

View full text Add to dashboard Cite

We consider a distributed learning problem in a wireless network, consisting of N distributed edge devices and a parameter server (PS). The objective function is a sum of the edge devices' local loss functions, who aim to train a shared model by communicating with the PS over multiple access channels (MAC). This problem has attracted a growing interest in distributed sensing systems, and more recently in federated learning, known as over-the-air computation. In this paper, we develop a novel Accelerated Gradient-descent Multiple Access (AGMA) algorithm that uses momentum-based gradient signals over noisy fading MAC to improve the convergence rate as compared to existing methods. Furthermore, AGMA does not require power control or beamforming to cancel the fading effect, which simplifies the implementation complexity. We analyze AGMA theoretically, and establish a finite-sample bound of the error for both convex and strongly convex loss functions with Lipschitz gradient. For the strongly convex case, we show that AGMA approaches the best-known linear convergence rate as the network increases. For the convex case, we show that AGMA significantly improves the sub-linear convergence rate as compared to existing methods. Finally, we present simulation results using real datasets that demonstrate better performance by AGMA.

show abstract

Section: B Main Resultsmentioning

confidence: 99%

Section: A Related Workmentioning

confidence: 99%

Section: The Proposed Accelerated Gradient-descent Multiple Access (A...mentioning

confidence: 99%

See 1 more Smart Citation

Accelerated Gradient Descent Learning over Multiple Access Fading Channels

Paul¹,

Friedman²,

Cohen³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…, (39) where {θ (s) } S s=1 are the samples produced by the CMC algorithm under evaluation. Specifically, as in [25], we consider multiple test functions, with each function given by one entry of the outer product matrix θθ T .…”

Section: Methodsmentioning

confidence: 99%

“…For large models, i.e., for large d, this requires large blocks in time/frequency and/or the use of sufficiently large antenna arrays. The integration of analog compression techniques (see, e.g., [39], [40]) with the wireless CMC strategies to be developed here can alleviate this problem and is left for future work. Finally, we assume that the channel matrices {H k } K k=1 have full rank with probability one.…”

Section: B Communication Modelmentioning

confidence: 99%

Channel-Driven Monte Carlo Sampling for Bayesian Distributed Learning in Wireless Data Centers

Liu¹,

Simeone²

2021

Preprint

View full text Add to dashboard Cite

Conventional frequentist learning, as assumed by existing federated learning protocols, is limited in its ability to quantify uncertainty, incorporate prior knowledge, guide active learning, and enable continual learning. Bayesian learning provides a principled approach to address all these limitations, at the cost of an increase in computational complexity. This paper studies distributed Bayesian learning in a wireless data center setting encompassing a central server and multiple distributed workers. Prior work on wireless distributed learning has focused exclusively on frequentist learning, and has introduced the idea of leveraging uncoded transmission to enable "over-the-air" computing. Unlike frequentist learning, Bayesian learning aims at evaluating approximations or samples from a global posterior distribution in the model parameter space. This work investigates for the first time the design of distributed one-shot, or "embarrassingly parallel", Bayesian learning protocols in wireless data centers via consensus Monte Carlo (CMC). Uncoded transmission is introduced not only as a way to implement "over-the-air" computing, but also as a mechanism to deploy channel-driven MC sampling: Rather than treating channel noise as a nuisance to be mitigated, channel-driven sampling utilizes channel noise as an integral part of the MC sampling process. A simple wireless CMC scheme is first proposed that is asymptotically optimal under Gaussian local posteriors. Then, for arbitrary local posteriors, a variational optimization strategy is introduced. Simulation results demonstrate that, if properly accounted for, channel noise can indeed contribute to MC sampling and does not necessarily decrease the accuracy level.

show abstract

Fundamentals and Preliminaries of Federated Learning

Chen,

Cui

2023

Wireless Networks

View full text Add to dashboard Cite

Analog Compression and Communication for Federated Learning over Wireless MAC

Cited by 20 publications

References 12 publications

Accelerated Gradient Descent Learning over Multiple Access Fading Channels

Accelerated Gradient Descent Learning over Multiple Access Fading Channels

Channel-Driven Monte Carlo Sampling for Bayesian Distributed Learning in Wireless Data Centers

Fundamentals and Preliminaries of Federated Learning

Contact Info

Product

Resources

About