Communication-Efficient Distributed Statistical Inference

Jordan, Michael I.; Lee, Jason D.; Yang, Y. Tony

doi:10.1080/01621459.2018.1429274

Cited by 355 publications

(382 citation statements)

References 18 publications

Supporting

Mentioning

381

Contrasting

Order By: Relevance

“…In return, downloading the models from servers to devices in proximity provision the latter intelligence to respond to real-time events. While computing speeds are growing rapidly, wireless transmission of high-dimensional data by many devices suffers from the scarcity of radio resources and hostility of wireless channels, resulting in a communication bottleneck for fast edge learning [5], [6].…”

Section: Introductionmentioning

confidence: 99%

Broadband Analog Aggregation for Low-Latency Federated Edge Learning

Zhu

Wang

Huang

2020

IEEE Trans. Wireless Commun.

685

489

View full text Add to dashboard Cite

To leverage rich data distributed at the network edge, a new machine-learning paradigm, called edge learning, has emerged where learning algorithms are deployed at the edge for providing intelligent services to mobile users.While computing speeds are advancing rapidly, the communication latency is becoming the bottleneck of fast edge learning. To address this issue, this work is focused on designing a low-latency multi-access scheme for edge learning. To this end, we consider a popular privacy-preserving framework, federated edge learning (FEEL), where a global AI-model at an edge-server is updated by aggregating (averaging) local models trained at edge devices. It is proposed that the updates simultaneously transmitted by devices over broadband channels should be analog aggregated "over-the-air" by exploiting the waveform-superposition property of a multi-access channel.Such broadband analog aggregation (BAA) results in dramatical communication-latency reduction compared with the conventional orthogonal access (i.e., OFDMA). In this work, the effects of BAA on learning performance are quantified targeting a single-cell random network. First, we derive two tradeoffs between communication-andlearning metrics, which are useful for network planning and optimization. The power control ("truncated channel inversion") required for BAA results in a tradeoff between the update-reliability [as measured by the receive signalto-noise ratio (SNR)] and the expected update-truncation ratio. Consider the scheduling of cell-interior devices to constrain path loss. This gives rises to the other tradeoff between the receive SNR and fraction of data exploited in learning. Next, the latency-reduction ratio of the proposed BAA with respect to the traditional OFDMA scheme is proved to scale almost linearly with the device population. Experiments based on a neural network and a real dataset are conducted for corroborating the theoretical results. In addition, we discuss the extensions of BAA to acquire safety against adversary attacks and integrate beamforming for enhancing cell-edge links.

show abstract

Section: Introductionmentioning

confidence: 99%

Broadband Analog Aggregation for Low-Latency Federated Edge Learning

Zhu

Wang

Huang

2020

IEEE Trans. Wireless Commun.

685

489

View full text Add to dashboard Cite

show abstract

“…This corollary shows that the quantized version of the unconstrained decentralized gradient method archives a similar upper bound as the projected gradient method in [3] (see (18)), even for N > 2. The only difference is that the log(d) factor in Equation (18) has been replaced with log(W ). The bound W does not always depend always depend on dimension d, e.g., the upper bound in Equation (10).…”

Section: B Main Result: Maintaining the Linear Convergencementioning

confidence: 99%

On Maintaining Linear Convergence of Distributed Learning and Optimization under Limited Communication

Magnússon

Shokri‐Ghadikolaei

2019

2019 53rd Asilomar Conference on Signals, Systems, and Computers

View full text Add to dashboard Cite

In distributed optimization and machine learning, multiple nodes coordinate to solve large problems. To do this, the nodes need to compress important algorithm information to bits so that it can be communicated over a digital channel. The communication time of these algorithms follows a complex interplay between a) the algorithm's convergence properties, b) the compression scheme, and c) the transmission rate offered by the digital channel. We explore these relationships for a general class of linearly convergent distributed algorithms. In particular, we illustrate how to design quantizers for these algorithms that compress the communicated information to a few bits while still preserving the linear convergence. Moreover, we characterize the communication time of these algorithms as a function of the available transmission rate. We illustrate our results on learning algorithms using different communication structures, such as decentralized algorithms where a single master coordinates information from many workers and fully distributed algorithms where only neighbours in a communication graph can communicate. We conclude that a co-design of machine learning and communication protocols are mandatory to flourish machine learning over networks.

show abstract

“…Shamir et al [37] and Zhang and Xiao [38] proposed truly communicationefficient distributed optimization algorithms which leveraged the local second-order information, though these approaches are only guaranteed to work for convex and smooth objectives. In a similar spirit, Wang et al [8], Jordan et al [9], and Ren et al [10] developed communication-efficient algorithms for sparse learning with 1 regularization. However, each of these works needs an assumption about the strong convexity of loss functions, which may limit their approaches to only a small set of real-world applications.…”

Section: B Related Workmentioning

confidence: 99%

“…III. THEORETICAL ANALYSIS Solving subproblem (3) is inspired by the approaches of Shamir et al [37], et al, Wang et al [8], and Jordan et al [9], and is designed to take advantage of both global first-order information and local higher-order information. Indeed, when ρ = 0 and L j is quadratic, (3) has the following closed form solution:…”

Section: Algorithmmentioning

confidence: 99%

Provably Communication-Efficient Asynchronous Distributed Inference for Convex and Nonconvex Problems

Ren

Haupt

2018

2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP)

View full text Add to dashboard Cite

This paper proposes and analyzes a communicationefficient distributed optimization framework for general nonconvex nonsmooth signal processing and machine learning problems under an asynchronous protocol. At each iteration, worker machines compute gradients of a known empirical loss function using their own local data, and a master machine solves a related minimization problem to update the current estimate. We prove that for nonconvex nonsmooth problems, the proposed algorithm converges with a sublinear rate over the number of communication rounds, coinciding with the best theoretical rate that can be achieved for this class of problems. Linear convergence is established without any statistical assumptions of the local data for problems characterized by composite loss functions whose smooth parts are strongly convex. Extensive numerical experiments verify that the performance of the proposed approach indeed improves -sometimes significantly -over other state-of-the-art algorithms in terms of total communication efficiency.

show abstract

Communication-Efficient Distributed Statistical Inference

Cited by 355 publications

References 18 publications

Broadband Analog Aggregation for Low-Latency Federated Edge Learning

Broadband Analog Aggregation for Low-Latency Federated Edge Learning

On Maintaining Linear Convergence of Distributed Learning and Optimization under Limited Communication

Provably Communication-Efficient Asynchronous Distributed Inference for Convex and Nonconvex Problems

Contact Info

Product

Resources

About