Some equivalences between Shannon entropy and Kolmogorov complexity

Leung-Yan-Cheong, S.; Cover, Thomas M.

doi:10.1109/tit.1978.1055891

Cited by 105 publications

(48 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, the stochastic complexity (6) as a model selection criterion is not a ected for large sample situation and mildly a ected otherwise under reparameterization. A nal problem is the term n ;1=4 in our criterion (6). From section 2 we know that it is obtained by an approximation.…”

Section: Discussionmentioning

confidence: 99%

Some notes on Rissanen's stochastic complexity

Qian

Künsch

1998

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

show abstract

Section: Discussionmentioning

confidence: 99%

Some notes on Rissanen's stochastic complexity

Qian

Künsch

1998

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

show abstract

“…The result then follows from Shannon's entropy lower bound of w i log 2 (1/w i ) on the average path length of any binary code [27]. The relation between this sum of logs of ranks and entropy is closely related to the efficiency of one-to-one codes in coding theory [9,23]; codes similar to the one shown in Figure 5 have also been used as part of more complex data compression schemes [6,13]. We note that an inequality in the other direction, …”

Section: Approximate Sum Of Subtree Clusteringmentioning

confidence: 98%

Squarepants in a tree

Eppstein

2009

ACM Trans. Algorithms

View full text Add to dashboard Cite

We provide efficient constant factor approximation algorithms for the problems of finding a hierarchical clustering of a point set in any metric space, minimizing the sum of minimimum spanning tree lengths within each cluster, and in the hyperbolic or Euclidean planes, minimizing the sum of cluster perimeters. Our algorithms for the hyperbolic and Euclidean planes can also be used to provide a pants decomposition, that is, a set of disjoint simple closed curves partitioning the plane minus the input points into subsets with exactly three boundary components, with approximately minimum total length. In the Euclidean case, these curves are squares; in the hyperbolic case, they combine our Euclidean square pants decomposition with our tree clustering method for general metric spaces.

show abstract

“…They are two sides of the same coin. The formal mathematical proof is rather subtle (see Kolmogorov, 1968;Zvonkin and Levin, 1970;Leung-Yan-Cheong and Cover, 1978;Zurek, 1989), but is intuitive and elicits even the most hardened theorists exclamations like "amazing" (Cover and Thomas, 2006;p. 463) and "beautiful" (Li and Vitanyi, 1997;p.…”

Section: An 'Amazing' and 'Beautiful' Factmentioning

confidence: 99%

Formal definitions of information and knowledge and their role in growth through structural change

Hilbert

2016

Structural Change and Economic Dynamics

View full text Add to dashboard Cite

The article provides a way to quantify the role of information and knowledge in growth through structural adjustments. The more is known about environmental patterns, the more growth can be obtained by redistributing resources accordingly among the evolving sectors (e.g. bet-hedging). Formal equations show that the amount of information about the environmental pattern is directly linked to the growth potential. This can be quantified by treating both information and knowledge formally through metrics like Shannon's mutual information and algorithmic Kolmogorov complexity from information theory and computer science. These mathematical metrics emerge naturally from our evolutionary equations. As such, information becomes a quantifiable ingredient of growth. The policy mechanism to convert information and knowledge into growth is structural adjustment. The presented approach is applied to the empirical case of U.S. export to showcase how information converts into growth potential.Keywords: evolutionary growth, information, international trade, bet hedging, fitness decomposition, knowledge. JEL codes: B25; B52; C02; D80; D81; D92; F14; G11; O43. The Nobel-laureate and co-founder of the Santa Fe Institute, Murray Gell-Mann, came to the conclusion that although complex systems "differ widely in their … attributes, they resemble one another in the way they handle information. That common feature is perhaps the best starting point for exploring how they operate" (1995, p. 21). The article uses formal definitions of information and knowledge from information theory and computer science, links them to related definitions from evolutionary economics, and showcases the how information can inform structural change in the economy. Information theoretic metrics of information naturally emerge from evolutionary decompositions of growth and can directly be linked to the growth potential due to the redistribution of resources (i.e. structural change of the economic population).An intuitive micro-economic example will set the stage and introduce the presented argument. Imagine a bakery that produces salty and sweet goods. If nothing else is known about the future environment, economic evolution would suggest to let market selection winnow out the more profitable among these two business options (Nelson and Winter, 1985). However, if we have information about future dynamics, it might be profitable to intervene into market selection. Recent big data analysis of statistical patterns has revealed that the demand for sweet goods grows with rain and the demand for salty goods with sunshine. This information about the environment allows to adjust the structure of its product offerings to the identified environmental pattern. The more it knows about the relation, the more growth potential. Being aware of this relationship and the environmental pattern with regard to rain and sunshine, productivity increments of up to 20 % have been reported for individual bakeries (Christensen, 2012). The potential to grow depends on what is kno...

show abstract

Some equivalences between Shannon entropy and Kolmogorov complexity

Cited by 105 publications

References 4 publications

Some notes on Rissanen's stochastic complexity

Some notes on Rissanen's stochastic complexity

Squarepants in a tree

Formal definitions of information and knowledge and their role in growth through structural change

Contact Info

Product

Resources

About