Understanding Interdependency Through Complex Information Sharing

Accurately determining dependency structure is critical to understanding a complex system's organization. We recently showed that the transfer entropy fails in a key aspect of this-measuring information flow-due to its conflation of dyadic and polyadic relationships. We extend this observation to demonstrate that Shannon information measures (entropy and mutual information, in their conditional and multivariate forms) can fail to accurately ascertain multivariate dependencies due to their conflation of qualitatively different relations among variables. This has broad implications, particularly when employing information to express the organization and mechanisms embedded in complex systems, including the burgeoning efforts to combine complex network theory with information theory. Here, we do not suggest that any aspect of information theory is wrong. Rather, the vast majority of its informational measures are simply inadequate for determining the meaningful relationships among variables within joint probability distributions. We close by demonstrating that such distributions exist across an arbitrary set of variables.

Section: Discussionmentioning

confidence: 99%

Multivariate Dependence beyond Shannon Information

James

Crutchfield

2017

“…They showed that, even though the PID atoms are among the very few measures that can distinguish between the two kinds of systems, a PID lattice with two source variables and one target variable cannot allot the full joint entropy H(X, Y, Z) of either system. The decomposition of the joint entropy in terms of information components that reflect qualitatively different interactions within the system has also been subject of recent research, that however relies on constructions differing substantially from the PID lattice [29,33].…”

Section: Preliminaries and State Of The Artmentioning

confidence: 99%

“…Understanding how information is distributed in trivariate systems should also provide a descriptive allotment of all parts of the joint entropy H(X, Y, Z) [21,33]. For comparison, Shannon's mutual information enables a semantic decomposition of the bivariate entropy H(X, Y) in terms of univariate conditional entropies and I(X : Y), that quantifies shared fluctuations (or covariations) between the two variables [33]:…”

Section: Decomposing the Joint Entropy Of A Trivariate Systemmentioning

confidence: 99%

Invariant Components of Synergy, Redundancy, and Unique Information among Three Variables

Pica

Piasini

Chicharro

et al. 2017

Abstract:In a system of three stochastic variables, the Partial Information Decomposition (PID) of Williams and Beer dissects the information that two variables (sources) carry about a third variable (target) into nonnegative information atoms that describe redundant, unique, and synergistic modes of dependencies among the variables. However, the classification of the three variables into two sources and one target limits the dependency modes that can be quantitatively resolved, and does not naturally suit all systems. Here, we extend the PID to describe trivariate modes of dependencies in full generality, without introducing additional decomposition axioms or making assumptions about the target/source nature of the variables. By comparing different PID lattices of the same system, we unveil a finer PID structure made of seven nonnegative information subatoms that are invariant to different target/source classifications and that are sufficient to describe the relationships among all PID lattices. This finer structure naturally splits redundant information into two nonnegative components: the source redundancy, which arises from the pairwise correlations between the source variables, and the non-source redundancy, which does not, and relates to the synergistic information the sources carry about the target. The invariant structure is also sufficient to construct the system's entropy, hence it characterizes completely all the interdependencies in the system.

“…The value of the predictive power will vary between 0% when X is not informative of Y and 100% when X is a perfect predictor of Y, i.e., X ≡ Y. In the literature, there has been numerous proposals for the use of information theory as a measure of temporal evolution [13]. Some are symmetric measures, such as the predictive information [12], and others enable, at the price of numerous acquisition time-points, to determine causal relationships between variables, such as the transfer entropy [14] or the Granger causality [15].…”

Section: Modeling a Shannon-like Communication Channel For Multicompomentioning

confidence: 99%

Multicomponent and Longitudinal Imaging Seen as a Communication Channel—An Application to Stroke

Giacalone

Frindel

Grenier

et al. 2017

Abstract:In longitudinal medical studies, multicomponent images of the tissues, acquired at a given stage of a disease, are used to provide information on the fate of the tissues. We propose a quantification of the predictive value of multicomponent images using information theory. To this end, we revisit the predictive information introduced for monodimensional time series and extend it to multicomponent images. The interest of this theoretical approach is illustrated on multicomponent magnetic resonance images acquired on stroke patients at acute and late stages, for which we propose an original and realistic model of noise together with a spatial encoding for the images. We address therefrom very practical questions such as the impact of noise on the predictability, the optimal choice of an observation scale and the predictability gain brought by the addition of imaging components.