Conservation of mutual and directed information

Massey, James L.; Massey, Philip

doi:10.1109/isit.2005.1523313

Cited by 138 publications

(99 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…3) Particularizing even further, we obtain whenever is the outcome of corrupting with additive noise, i.e., , where and are independent. 4) The fifth part of the proposition can be considered the continuous-time analogue of the discrete-time conservation law [22] (66)…”

Section: Remarksmentioning

confidence: 99%

Directed Information, Causal Estimation, and Communication in Continuous Time

Weissman

Kim

Permuter

2013

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

A notion of directed information between two continuous-time processes is proposed. A key component in the definition is taking an infimum over all possible partitions of the time interval, which plays a role no less significant than the supremum over "space" partitions inherent in the definition of mutual information. Properties and operational interpretations in estimation and communication are then established for the proposed notion of directed information. For the continuous-time additive white Gaussian noise channel, it is shown that Duncan's classical relationship between causal estimation error and mutual information continues to hold in the presence of feedback upon replacing mutual information by directed information. A parallel result is established for the Poisson channel. The utility of this relationship is demonstrated in computing the directed information rate between the input and output processes of a continuous-time Poisson channel with feedback, where the channel input process is constrained to be constant between events at the channel output. Finally, the capacity of a wide class of continuous-time channels with feedback is established via directed information, characterizing the fundamental limit on reliable communication.

show abstract

Section: Remarksmentioning

confidence: 99%

Directed Information, Causal Estimation, and Communication in Continuous Time

Weissman

Kim

Permuter

2013

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

show abstract

“…The works originate from the ideas of Granger (1969) on causal interactions, and was placed in an information-theoretic perspective by Massey (1990), Massey and Massey (2005) and Schreiber (2000). Amblard and Michel (2011) here merge the two ideas and extract Granger causality graphs by using directed information measures.…”

Section: Current Statementioning

confidence: 99%

Information theory in neuroscience

2011

View full text Add to dashboard Cite

“…It is important to notice that cues from the past can still tell us something about the upcoming present (additionally to the present cue). Permuter, Young-Han and Weissman (2011) have shown that instead of Shannon's mutual information, the correct measure in this more general case is Massey's directed information ( → ) (Massey, 1990;Massey and Massey, 2005;also Kramer, 1998). If the present environmental state and the side information are pairwise independent (e.g.…”

Section: The Fitness Value Of Informationmentioning

confidence: 99%

Formal definitions of information and knowledge and their role in growth through structural change

Hilbert

2016

Structural Change and Economic Dynamics

View full text Add to dashboard Cite

The article provides a way to quantify the role of information and knowledge in growth through structural adjustments. The more is known about environmental patterns, the more growth can be obtained by redistributing resources accordingly among the evolving sectors (e.g. bet-hedging). Formal equations show that the amount of information about the environmental pattern is directly linked to the growth potential. This can be quantified by treating both information and knowledge formally through metrics like Shannon's mutual information and algorithmic Kolmogorov complexity from information theory and computer science. These mathematical metrics emerge naturally from our evolutionary equations. As such, information becomes a quantifiable ingredient of growth. The policy mechanism to convert information and knowledge into growth is structural adjustment. The presented approach is applied to the empirical case of U.S. export to showcase how information converts into growth potential.Keywords: evolutionary growth, information, international trade, bet hedging, fitness decomposition, knowledge. JEL codes: B25; B52; C02; D80; D81; D92; F14; G11; O43. The Nobel-laureate and co-founder of the Santa Fe Institute, Murray Gell-Mann, came to the conclusion that although complex systems "differ widely in their … attributes, they resemble one another in the way they handle information. That common feature is perhaps the best starting point for exploring how they operate" (1995, p. 21). The article uses formal definitions of information and knowledge from information theory and computer science, links them to related definitions from evolutionary economics, and showcases the how information can inform structural change in the economy. Information theoretic metrics of information naturally emerge from evolutionary decompositions of growth and can directly be linked to the growth potential due to the redistribution of resources (i.e. structural change of the economic population).An intuitive micro-economic example will set the stage and introduce the presented argument. Imagine a bakery that produces salty and sweet goods. If nothing else is known about the future environment, economic evolution would suggest to let market selection winnow out the more profitable among these two business options (Nelson and Winter, 1985). However, if we have information about future dynamics, it might be profitable to intervene into market selection. Recent big data analysis of statistical patterns has revealed that the demand for sweet goods grows with rain and the demand for salty goods with sunshine. This information about the environment allows to adjust the structure of its product offerings to the identified environmental pattern. The more it knows about the relation, the more growth potential. Being aware of this relationship and the environmental pattern with regard to rain and sunshine, productivity increments of up to 20 % have been reported for individual bakeries (Christensen, 2012). The potential to grow depends on what is kno...

show abstract

Conservation of mutual and directed information

Cited by 138 publications

References 2 publications

Directed Information, Causal Estimation, and Communication in Continuous Time

Directed Information, Causal Estimation, and Communication in Continuous Time

Information theory in neuroscience

Formal definitions of information and knowledge and their role in growth through structural change

Contact Info

Product

Resources

About