No abstract

The sum-product or belief propagation (BP) algorithm is a widely-used messagepassing algorithm for computing marginal distributions in graphical models with discrete variables. At the core of the BP message updates, when applied to a graphical model with pairwise interactions, lies a matrix-vector product with complexity that is quadratic in the state dimension d, and requires transmission of a (d − 1)-dimensional vector of real numbers (messages) to its neighbors. Since various applications involve very large state dimensions, such computation and communication complexities can be prohibitively complex. In this paper, we propose a low-complexity variant of BP, referred to as stochastic belief propagation (SBP). As suggested by the name, it is an adaptively randomized version of the BP message updates in which each node passes randomly chosen information to each of its neighbors. The SBP message updates reduce the computational complexity (per iteration) from quadratic to linear in d, without assuming any particular structure of the potentials, and also reduce the communication complexity significantly, requiring only log d bits transmission per edge. Moreover, we establish a number of theoretical guarantees for the performance of SBP, showing that it converges almost surely to the BP fixed point for any tree-structured graph, and for graphs with cycles satisfying a contractivity condition. In addition, for these graphical models, we provide non-asymptotic upper bounds on the convergence rate, showing that the ℓ ∞ norm of the error vector decays no slower than O 1/ √ t with the number of iterations t on trees and the mean square error decays as O 1/t for general graphs. These analysis show that SBP can provably yield reductions in computational and communication complexities for various classes of graphical models.

No abstract

The problem of network-constrained averaging is to compute the average of a set of values distributed throughout a graph G using an algorithm that can pass messages only along graph edges. We study this problem in the noisy setting, in which the communication along each link is modeled by an additive white Gaussian noise channel. We propose a two-phase decentralized algorithm, and we use stochastic approximation methods in conjunction with the spectral graph theory to provide concrete (non-asymptotic) bounds on the mean-squared error. Having found such bounds, we analyze how the number of iterations T G (n; δ) required to achieve mean-squared error δ scales as a function of the graph topology and the number of nodes n. Previous work provided guarantees with the number of iterations scaling inversely with the second smallest eigenvalue of the Laplacian. This paper gives an algorithm that reduces this graph dependence to the graph diameter, which is the best scaling possible. I. INTRODUCTIONThe problem of network-constrained averaging is to compute the average of a set of numbers distributed throughout a network, using an algorithm that is allowed to pass messages only along edges of the graph.Motivating applications include sensor networks, in which individual motes have limited memory and communication ability, and massive databases and server farms, in which memory constraints preclude storing all data at a central location. In typical applications, the average might represent a statistical estimate of some physical quantity (e.g., temperature, pressure etc.), or an intermediate quantity in a more complex algorithm (e.g., for distributed optimization). There is now an extensive literature on network-averaging, consensus problems, as well as distributed optimization and estimation (e.g., see the papers [7], [12], [10], [30], [20], [3], [4], [8], [23], [22]). The bulk of the earlier work has focused on the noiseless variant, in which communication between nodes in the graph is assumed to be noiseless. A more recent line of work has studied versions of the problem with noisy communication links (e.g., see the papers [18], [15], [27], [2], [29], [19], [24] and references therein).The focus of this paper is a noisy version of network-constrained averaging in which inter-node communication is modeled by an additive white Gaussian noise (AWGN) channel. Given this randomness, any algorithm is necessarily stochastic, and the corresponding sequence of random variables can be analyzed in various ways. The simplest question to ask is whether the algorithm is consistent-that is, does it compute an approximate average or achieve consensus in an asymptotic sense for a given fixed graph? A more refined analysis seeks to provide information about this convergence rate. In this paper, we do so by posing the following question: for a given algorithm, how does number of iterations required to compute the average to within δ-accuracy scale as a function of the graph topology and number of nodes n? For obvious reasons, we refer to th...

Belief propagation (BP) is a widely used algorithm for computing the marginal distributions in graphical models. However, in applications involving continuous random variables, the messages themselves are real-valued functions, which leads to significant computational bottlenecks. In this paper, we propose a low complexity method for performing belief propagation for continuous state space problems. Our algorithm, which we refer to as quantized stochastic belief propagation (QSBP), is a randomized variant of BP in which each node only passes stochastically chosen information at each round. The most attractive feature of QSBP is its significant gain in computational and communication efficiencies. In addition, we provide some theoretical guarantees including almost sure convergence and the rate of convergence for the case of tree-structured graphical models.