Learning Data Dependency with Communication Cost

Abstract

In this paper, we consider the problem of recovering a graph that represents the statistical data dependency among nodes for a set of data samples generated by nodes, which provides the basic structure to perform an inference task, such as MAP (maximum a posteriori). This problem is referred to as structure learning. When nodes are spatially separated in different locations, running an inference algorithm requires a non-negligible amount of message passing, incurring some communication cost. We inevitably have the trade-off between the accuracy of structure learning and the cost we need to pay to perform a given message-passing based inference task because the learnt edge structures of data dependency and physical connectivity graph are often highly different. In this paper, we formalize this trade-off in an optimization problem which outputs the data dependency graph that jointly considers learning accuracy and message-passing costs. We focus on a distributed MAP as the target inference task due to its popularity, and consider two different implementations, ASYNC-MAP and SYNC-MAP that have different message-passing mechanisms and thus different cost structures. In ASYNC-MAP, we propose a polynomial time learning algorithm that is optimal, motivated by the problem of finding a maximum weight spanning tree. In SYNC-MAP, we first prove that it is NP-hard and propose a greedy heuristic. For both implementations, we then quantify how the probability that the resulting data graphs from those learning algorithms differ from the ideal data graph decays as the number of data samples grows, using the large deviation principle, where the decaying rate is characterized by some topological structures of both original data dependency and physical connectivity graphs as well as the degree of the trade-off, which provides some guideline on how many samples are necessary to obtain a certain learning accuracy. We validate our theoretical findings through extensive simulations, which confirms that it has a good match.

Decentralized Learning of Tree-Structured Gaussian Graphical Models from Noisy Data

“…The error exponent of Chow-Liu algorithm is analyzed in [8], [9], and [23] for tree-structured discrete distribution, continuous distribution, and tree-structured Ising model with side information, respectively. The tree-structured graphical models are learned in the decentralized system [10] with Gaussian random variables and the fully distributed system [15] with binary random variables under communication constraints. Tavassolipour et al study the structure learning problem where the data set is distributed vertically across machines [10].…”

“…Our proposed bounds are evaluated on synthetic data sets, and similar data sets have been used in literature such as [9]- [15], [18].…”