Information projections revisited

Csiszár, Imre; Matúš, František

doi:10.1109/isit.2000.866788

Cited by 73 publications

(140 citation statements)

References 3 publications

Supporting

Mentioning

136

Contrasting

Order By: Relevance

“…Furthermore, since is the edge set of the tree distribution , the optimization for the ML tree distribution reduces to the MWST search for the optimal edge set as in (13).…”

Section: Theorem 1 (Chow-liu Tree Learning [3])mentioning

confidence: 99%

See 2 more Smart Citations

A Large-Deviation Analysis of the Maximum-Likelihood Learning of Markov Tree Structures

Tan

Anandkumar

Tong

et al. 2011

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Abstract-The problem of maximum-likelihood (ML) estimation of discrete tree-structured distributions is considered. Chow and Liu established that ML-estimation reduces to the construction of a maximum-weight spanning tree using the empirical mutual information quantities as the edge weights. Using the theory of large-deviations, we analyze the exponent associated with the error probability of the event that the ML-estimate of the Markov tree structure differs from the true tree structure, given a set of independently drawn samples. By exploiting the fact that the output of ML-estimation is a tree, we establish that the error exponent is equal to the exponential rate of decay of a single dominant crossover event. We prove that in this dominant crossover event, a non-neighbor node pair replaces a true edge of the distribution that is along the path of edges in the true tree graph connecting the nodes in the non-neighbor pair. Using ideas from Euclidean information theory, we then analyze the scenario of ML-estimation in the very noisy learning regime and show that the error exponent can be approximated as a ratio, which is interpreted as the signal-to-noise ratio (SNR) for learning tree distributions. We show via numerical experiments that in this regime, our SNR approximation is accurate.Index Terms-Error exponent, Euclidean information theory, large-deviations principle, Markov structure, maximum-likelihood (ML) distribution estimation, tree-structured distributions.

show abstract

“…Furthermore, since is the edge set of the tree distribution , the optimization for the ML tree distribution reduces to the MWST search for the optimal edge set as in (13).…”

Section: Theorem 1 (Chow-liu Tree Learning [3])mentioning

confidence: 99%

“…For instance, for a star graph, the diameter . For a balanced tree, 13 ; hence, the number of computations is .…”

Section: Theorem 7 (Computational Complexity For )mentioning

confidence: 99%

See 1 more Smart Citation

A Large-Deviation Analysis of the Maximum-Likelihood Learning of Markov Tree Structures

Tan

Anandkumar

Tong

et al. 2011

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

show abstract

“…If t * is unique, then it is called the reverse I-projection [12] (Section I.A) of t onto P M . Since t * ∈ P M , its variational distance optimal approximation t vd has the same support as t, which allows us to bound…”

Section: M-type Approximation Minimizing D(t P)mentioning

confidence: 99%

“…Reverse I-projections admit a Pythagorean inequality [12] (Theorem 1). In other words, if p is a distribution, p * its reverse I-projection onto a set S, and q any distribution in S, then…”

Section: Appendix A2 Proof Of Lemmamentioning

confidence: 99%

Greedy Algorithms for Optimal Distribution Approximation

Geiger

Böcherer

2016

Entropy

View full text Add to dashboard Cite

Abstract:The approximation of a discrete probability distribution t by an M-type distribution p is considered. The approximation error is measured by the informational divergence D(t p), which is an appropriate measure, e.g., in the context of data compression. Properties of the optimal approximation are derived and bounds on the approximation error are presented, which are asymptotically tight. A greedy algorithm is proposed that solves this M-type approximation problem optimally. Finally, it is shown that different instantiations of this algorithm minimize the informational divergence D(p t) or the variational distance p − t 1 .

show abstract

A Reaction Network Scheme Which Implements Inference and Learning for Hidden Markov Models

Singh

Wiuf

Behera

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

With a view towards molecular communication systems and molecular multi-agent systems, we propose the Chemical Baum-Welch Algorithm, a novel reaction network scheme that learns parameters for Hidden Markov Models (HMMs). Each reaction in our scheme changes only one molecule of one species to one molecule of another. The reverse change is also accessible but via a different set of enzymes, in a design reminiscent of futile cycles in biochemical pathways. We show that every fixed point of the Baum-Welch algorithm for HMMs is a fixed point of our reaction network scheme, and every positive fixed point of our scheme is a fixed point of the Baum-Welch algorithm. We prove that the "Expectation" step and the "Maximization" step of our reaction network separately converge exponentially fast. We simulate mass-action kinetics for our network on an example sequence, and show that it learns the same parameters for the HMM as the Baum-Welch algorithm.

show abstract

Information projections revisited

Cited by 73 publications

References 3 publications

A Large-Deviation Analysis of the Maximum-Likelihood Learning of Markov Tree Structures

A Large-Deviation Analysis of the Maximum-Likelihood Learning of Markov Tree Structures

Greedy Algorithms for Optimal Distribution Approximation

A Reaction Network Scheme Which Implements Inference and Learning for Hidden Markov Models

Contact Info

Product

Resources

About