Shared Information—New Insights and Problems in Decomposing Information in Complex Systems

Abstract

How can the information that a set {X1, . . . , Xn} of random variables contains about another random variable S be decomposed? To what extent do different subgroups provide the same, i.e. shared or redundant, information, carry unique information or interact for the emergence of synergistic information? Recently Williams and Beer proposed such a decomposition based on natural properties for shared information. While these properties fix the structure of the decomposition, they do not uniquely specify the values of the different terms. Therefore, we investigate additional properties such as strong symmetry and left monotonicity. We find that strong symmetry is incompatible with the properties proposed by Williams and Beer. Although left monotonicity is a very natural property for an information measure it is not fulfilled by any of the proposed measures. We also study a geometric framework for information decompositions and ask whether it is possible to represent shared information by a family of posterior distributions. Finally, we draw connections to the notions of shared knowledge and common knowledge in game theory. While many people believe that independent variables cannot share information, we show that in game theory independent agents can have shared knowledge, but not common knowledge. We conclude that intuition and heuristic arguments do not suffice when arguing about information.

Quantifying Redundant Information in Predicting a Target Random Variable

“…There are a number of intuitive properties, proposed in [5,[9][10][11][12][13], that are considered desirable for the intersection information measure I ∩ to satisfy:…”

“…However, it has been challenging [9][10][11][12] to come up with precise information-theoretic definitions of synergy and redundancy that are consistent with all intuitively desired properties.…”

On Extractable Shared Information

“…A series of recent papers have focused on the bivariate information decomposition problem [1][2][3][4][5][6]. Consider three random variables S, X 1 , X 2 with finite alphabets S, X 1 and X 2 , respectively.…”

“…While their measure has subsequently been criticized for "not measuring the right thing" [4][5][6], there has been no successful attempt to find better measures, except for the bivariate case (k = 2) [1,4]. One problem seems to be the lack of a clear consensus on what an ideal measure of shared (or unique or complementary) information should look like and what properties it should satisfy.…”

“…In particular, the Williams-Beer axioms only put crude bounds on the values of the functions SI, U I and CI. Therefore, additional axioms have been proposed by various authors [4][5][6]. Unfortunately, some of these properties contradict each other [5], and the question for the right axiomatic characterization is still open.…”