Non-Asymptotic Analysis of an Optimal Algorithm for Network-Constrained Averaging With Noisy Links

Abstract

The problem of network-constrained averaging is to compute the average of a set of values distributed throughout a graph G using an algorithm that can pass messages only along graph edges. We study this problem in the noisy setting, in which the communication along each link is modeled by an additive white Gaussian noise channel. We propose a two-phase decentralized algorithm, and we use stochastic approximation methods in conjunction with the spectral graph theory to provide concrete (non-asymptotic) bounds on the mean-squared error. Having found such bounds, we analyze how the number of iterations T G (n; δ) required to achieve mean-squared error δ scales as a function of the graph topology and the number of nodes n. Previous work provided guarantees with the number of iterations scaling inversely with the second smallest eigenvalue of the Laplacian. This paper gives an algorithm that reduces this graph dependence to the graph diameter, which is the best scaling possible. I. INTRODUCTIONThe problem of network-constrained averaging is to compute the average of a set of numbers distributed throughout a network, using an algorithm that is allowed to pass messages only along edges of the graph.Motivating applications include sensor networks, in which individual motes have limited memory and communication ability, and massive databases and server farms, in which memory constraints preclude storing all data at a central location. In typical applications, the average might represent a statistical estimate of some physical quantity (e.g., temperature, pressure etc.), or an intermediate quantity in a more complex algorithm (e.g., for distributed optimization). There is now an extensive literature on network-averaging, consensus problems, as well as distributed optimization and estimation (e.g., see the papers [7], [12], [10], [30], [20], [3], [4], [8], [23], [22]). The bulk of the earlier work has focused on the noiseless variant, in which communication between nodes in the graph is assumed to be noiseless. A more recent line of work has studied versions of the problem with noisy communication links (e.g., see the papers [18], [15], [27], [2], [29], [19], [24] and references therein).The focus of this paper is a noisy version of network-constrained averaging in which inter-node communication is modeled by an additive white Gaussian noise (AWGN) channel. Given this randomness, any algorithm is necessarily stochastic, and the corresponding sequence of random variables can be analyzed in various ways. The simplest question to ask is whether the algorithm is consistent-that is, does it compute an approximate average or achieve consensus in an asymptotic sense for a given fixed graph? A more refined analysis seeks to provide information about this convergence rate. In this paper, we do so by posing the following question: for a given algorithm, how does number of iterations required to compute the average to within δ-accuracy scale as a function of the graph topology and number of nodes n? For obvious reasons, we refer to th...

Information-Theoretic Lower Bounds for Distributed Function Computation

“…Moreover, from (A.15) and (A. 16) we see that the random objects W Si , X T Si , Y T Si : i ∈ {1, . .…”

“…Here and below, the estimates for a network of bidirectional BSCs are obtained using the bounds (16) and (17).…”

Generalized geometric programming for rate allocation in consensus

“…In realistic scenarios, the nodes must communicate within bandwidth and energy constraints, which can have a significant impact on the convergence of distributed averaging algorithms. Although many papers have been published on quantized consensus in recent years (e.g., [1,4,[11][12][13][14][15][16][17]), 2 a large portion considers trade-offs among run time, communication load, and final accuracy without formulating the problem as constrained optimization. Early publications on quantized consensus (e.g., [11,19]) show that introducing perturbations of constant variance (such as quantization error) into the traditional consensus state update prevents convergence due to the limited precision of the quantizer.…”

“…In many distributed computing settings, it is necessary to compute a function of data that may be dispersed among a number of computing nodes. Examples include wireless sensor networks (WSNs), where each agent observes a different measurement of a physical process, and largescale server farms, where the size of the data set requires distributed storage [1]. The class of distributed algorithms considered in this paper computes these functions using only interactions among local subsets of the network nodes.…”

“…As a motivating example for our lossy compression framework, we consider the application of distributed averaging in representation learning for the internet of things (IoT). Distributed average consensus helps different nodes share intermediate results within a distributed version of the power method [6], 1 which can be used in distributed dictionary learning schemes such as cloud K-SVD [5]. Take face recognition as an example.…”