Greedy Learning of Graphical Models with Small Girth

Ray, Avik; Sanghavi, Sujay; Shakkottai, Sanjay

doi:10.21236/ada599141

Cited by 6 publications

(15 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A famous early example of such an algorithmic result is due to Chow and Liu from 1968 [CL68] who gave an efficient algorithm for learning graphical models where the underlying graph is a tree. Subsequent work considered generalizations of trees [ATHW11] and graphs under various strong assumptions (e.g., restricted strong convexity [NRWY10] or correlation decay [BMS13,RSS12]).…”

Section: Introductionmentioning

confidence: 99%

Learning Graphical Models Using Multiplicative Weights

Klivans

Meka²

2017

2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS)

120

View full text Add to dashboard Cite

We give a simple, multiplicative-weight update algorithm for learning undirected graphical models or Markov random fields (MRFs). The approach is new, and for the well-studied case of Ising models or Boltzmann machines, we obtain an algorithm that uses a nearly optimal number of samples and has running timeÕ(n 2 ) (where n is the dimension), subsuming and improving on all prior work. Additionally, we give the first efficient algorithm for learning Ising models over non-binary alphabets.Our main application is an algorithm for learning the structure of t-wise MRFs with nearlyoptimal sample complexity (up to polynomial losses in necessary terms that depend on the weights) and running time that is n O(t) . In addition, given n O(t) samples, we can also learn the parameters of the model and generate a hypothesis that is close in statistical distance to the true MRF. All prior work runs in time n Ω(d) for graphs of bounded degree d and does not generate a hypothesis close in statistical distance even for t = 3. We observe that our runtime has the correct dependence on n and t assuming the hardness of learning sparse parities with noise.Our algorithm-the Sparsitron-is easy to implement (has only one parameter) and holds in the on-line setting. Its analysis applies a regret bound from Freund and Schapire's classic Hedge algorithm. It also gives the first solution to the problem of learning sparse Generalized Linear Models (GLMs).

show abstract

Section: Introductionmentioning

confidence: 99%

Learning Graphical Models Using Multiplicative Weights

Klivans

Meka²

2017

2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS)

120

View full text Add to dashboard Cite

show abstract

“…The summations indexed by j / ∈ {u, v} are over nodes in the size d+1 clique under consideration. The last inequality follows by observing that the largest value is achieved in (22) when σ (l−1) v = −1 and j / ∈{u,v} σ (l−1) j → −∞. By symmetry the same bound holds for the ratio of conditional probabilities of σ u = −1.…”

Section: A Bound On Kl Divergencementioning

confidence: 81%

“…It was first observed in [6] that it is possible to efficiently learn models with (exponential) decay of correlations, under the additional assumption that neighboring variables have correlation bounded away from zero. A variety of other papers including [21], [22], [23], [24] give alternative low-complexity algorithms, but also require the CDP. A number of structure learning algorithms are based on convex optimization, such as Ravikumar et al's [25] approach using regularized node-wise logistic regression.…”

Section: A Complexity Of Graphical Model Learningmentioning

confidence: 99%

Learning graphical models from the Glauber dynamics

Bresler

Gamarnik

Shah

2014

2014 52nd Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

Abstract-In this paper we consider the problem of learning undirected graphical models from data generated according to the Glauber dynamics. The Glauber dynamics is a Markov chain that sequentially updates individual nodes (variables) in a graphical model and it is frequently used to sample from the stationary distribution (to which it converges given sufficient time). Additionally, the Glauber dynamics is a natural dynamical model in a variety of settings. This work deviates from the standard formulation of graphical model learning in the literature, where one assumes access to i.i.d. samples from the distribution.Much of the research on graphical model learning has been directed towards finding algorithms with low computational cost. As the main result of this work, we establish that the problem of reconstructing binary pairwise graphical models is computationally tractable when we observe the Glauber dynamics. Specifically, we show that a binary pairwise graphical model on p nodes with maximum degree d can be learned in time f (d)p 2 log p, for a function f (d), using nearly the information-theoretic minimum number of samples.

show abstract

“…• From (19), the KL divergence from a single-edge graph to the empty graph is upper bounded by λ tanh λ. Using this fact along with (17), any graph in T has a KL divergence to the empty graph of at most = αλ tanh λ. Combining these with (12) gives the necessary condition…”

Section: Ensemble1(α) [Isolated Edges Ensemble]mentioning

confidence: 99%

“…• The total number of possible edges is α m 2 , and hence the total number of graphs is |T | = 2 α( m 2 ) . • The maximal degree of each graph is at most m − 1. due to (17). Substituting these into (12), setting q max = θ 2 α m 2 for some θ 2 ∈ 0, 1 2 , and applying some simplifications, we obtain the following necessary condition for P e (q max ) ≤ δ:…”

Section: Ensemble1(α) [Isolated Edges Ensemble]mentioning

confidence: 99%

On the Difficulty of Selecting Ising Models with Approximate Recovery

Scarlett

Cevher

2016

IEEE Trans. on Signal and Inf. Process. over Networks

View full text Add to dashboard Cite

Abstract-In this paper, we consider the problem of estimating the underlying graph associated with an Ising model given a number of independent and identically distributed samples. We adopt an approximate recovery criterion that allows for a number of missed edges or incorrectly-included edges, in contrast with the widely-studied exact recovery problem. Our main results provide information-theoretic lower bounds on the sample complexity for graph classes imposing constraints on the number of edges, maximal degree, and other properties. We identify a broad range of scenarios where, either up to constant factors or logarithmic factors, our lower bounds match the best known lower bounds for the exact recovery criterion, several of which are known to be tight or near-tight. Hence, in these cases, approximate recovery has a similar difficulty to exact recovery in the minimax sense.Our bounds are obtained via a modification of Fano's inequality for handling the approximate recovery criterion, along with suitably-designed ensembles of graphs that can broadly be classed into two categories: (i) Those containing graphs that contain several isolated edges or cliques and are thus difficult to distinguish from the empty graph; (ii) Those containing graphs for which certain groups of nodes are highly correlated, thus making it difficult to determine precisely which edges connect them. We support our theoretical results on these ensembles with numerical experiments.

show abstract

Greedy Learning of Graphical Models with Small Girth

Cited by 6 publications

References 7 publications

Learning Graphical Models Using Multiplicative Weights

Learning Graphical Models Using Multiplicative Weights

Learning graphical models from the Glauber dynamics

On the Difficulty of Selecting Ising Models with Approximate Recovery

Contact Info

Product

Resources

About