Ordered Gradient Approach for Communication-Efficient Distributed Learning

Chen, Yi-Cheng; Sadler, Brian M.; Blum, Rick S.

doi:10.1109/spawc48557.2020.9153887

Cited by 9 publications

(4 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The second class of approaches focuses on the reduction of the communication iterations by eliminating the communication between some of the workers and the master node in some iterations [16]. The work [16] has proposed lazily aggregated gradient (LAG) for communication-efficient distributed learning in master-worker architectures. In LAG, each worker reports its gradient vector to the master node only if the gradient changes from the last communication iteration are large enough.…”

Section: A Literature Surveymentioning

confidence: 99%

FedCau: A Proactive Stop Policy for Communication and Computation Efficient Federated Learning

Mahmoudi,

Ghadikolaei,

Barros Da Silva

et al. 2024

IEEE Trans. Wireless Commun.

View full text Add to dashboard Cite

This paper investigates efficient distributed training of a Federated Learning (FL) model over a wireless network of wireless devices. The communication iterations of the distributed training algorithm may be substantially deteriorated or even blocked by the effects of the devices' background traffic, packet losses, congestion, or latency. We abstract the communicationcomputation impacts as an 'iteration cost' and propose a costaware causal FL algorithm (FedCau) to tackle this problem. We propose an iteration-termination method that trade-offs the training performance and networking costs. We apply our approach when workers use the slotted-ALOHA, carrier-sense multiple access with collision avoidance (CSMA/CA), and orthogonal frequency-division multiple access (OFDMA) protocols. We show that, given a total cost budget, the training performance degrades as either the background communication traffic or the dimension of the training problem increases. Our results demonstrate the importance of proactively designing optimal cost-efficient stopping criteria to avoid unnecessary communication-computation costs to achieve a marginal FL training improvement. We validate our method by training and testing FL over the MNIST and CIFAR-10 dataset. Finally, we apply our approach to existing communication efficient FL methods from the literature, achieving further efficiency. We conclude that cost-efficient stopping criteria are essential for the success of practical FL over wireless networks.

show abstract

Section: A Literature Surveymentioning

confidence: 99%

FedCau: A Proactive Stop Policy for Communication and Computation Efficient Federated Learning

Mahmoudi,

Ghadikolaei,

Barros Da Silva

et al. 2024

IEEE Trans. Wireless Commun.

View full text Add to dashboard Cite

show abstract

“…Censorship in distributed learning reduces communication, but some useful information may be lost. [127] studies an ordered gradient method that uses sorting to eliminate some of the worker-to-server upstream communication typically required in gradient descent methods. [128] and [129] study gradient coding to reduce communication costs while being able to reduce the latency caused by slow-running machines.…”

Section: A Communication Costmentioning

confidence: 99%

Distributed Learning for Wireless Communications: Methods, Applications and Challenges

Qian

Yang

Xiao

et al. 2022

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

With its privacy-preserving and decentralized features, distributed learning plays an irreplaceable role in the era of wireless networks with a plethora of smart terminals, an explosion of information volume and increasingly sensitive data privacy issues. There is a tremendous increase in the number of scholars investigating how distributed learning can be employed to emerging wireless network paradigms in the physical layer, media access control layer and network layer. Nonetheless, researches on distributed learning for wireless communications are still in its infancy. In this paper, we review the contemporary technical applications of distributed learning for wireless communications. We first introduce the typical frameworks and algorithms for distributed learning. Examples of applications of distributed learning frameworks in the emerging wireless network paradigms are then provided. Finally, main research directions and challenges of distributed learning for wireless communications are discussed.

show abstract

“…To the best of our knowledge, ordered transmissions have not been applied to federated learning in a completely distributed setting. Some extensions to the work in [21] have been developed, including the application of ordering to quickest change detection in sensor networks [22], nearest-neighbor learning [23], and ordered gradient descent (GD) in a worker-server architecture setting [24].…”

Section: Introductionmentioning

confidence: 99%

Communication Efficient Federated Learning via Ordered ADMM in a Fully Decentralized Setting

Chen¹,

Blum²,

Sadler³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The challenge of communication-efficient distributed optimization has attracted attention in recent years. In this paper, a communication efficient algorithm, called orderingbased alternating direction method of multipliers (OADMM) is devised in a general fully decentralized network setting where a worker can only exchange messages with neighbors. Compared to the classical ADMM, a key feature of OADMM is that transmissions are ordered among workers at each iteration such that a worker with the most informative data broadcasts its local variable to neighbors first, and neighbors who have not transmitted yet can update their local variables based on that received transmission. In OADMM, we prohibit workers from transmitting if their current local variables are not sufficiently different from their previously transmitted value. A variant of OADMM, called SOADMM, is proposed where transmissions are ordered but transmissions are never stopped for each node at each iteration. Numerical results demonstrate that given a targeted accuracy, OADMM can significantly reduce the number of communications compared to existing algorithms including ADMM. We also show numerically that SOADMM can accelerate convergence, resulting in communication savings compared to the classical ADMM.

show abstract

Ordered Gradient Approach for Communication-Efficient Distributed Learning

Cited by 9 publications

References 20 publications

FedCau: A Proactive Stop Policy for Communication and Computation Efficient Federated Learning

FedCau: A Proactive Stop Policy for Communication and Computation Efficient Federated Learning

Distributed Learning for Wireless Communications: Methods, Applications and Challenges

Communication Efficient Federated Learning via Ordered ADMM in a Fully Decentralized Setting

Contact Info

Product

Resources

About