Pointwise Partial Information Decomposition Using the Specificity and Ambiguity Lattices

Abstract

What are the distinct ways in which a set of predictor variables can provide information about a target variable? When does a variable provide unique information, when do variables share redundant information, and when do variables combine synergistically to provide complementary information? The redundancy lattice from the partial information decomposition of Williams and Beer provided a promising glimpse at the answer to these questions. However, this structure was constructed using a much criticised measure of redundant information, and despite sustained research, no completely satisfactory replacement measure has been proposed. In this paper, we take a different approach, applying the axiomatic derivation of the redundancy lattice to a single realisation from a set of discrete variables. To overcome the difficulty associated with signed pointwise mutual information, we apply this decomposition separately to the unsigned entropic components of pointwise mutual information which we refer to as the specificity and ambiguity. This yields a separate redundancy lattice for each component. Then based upon an operational interpretation of redundancy, we define measures of redundant specificity and ambiguity enabling us to evaluate the partial information atoms in each lattice. These atoms can be recombined to yield the sought-after multivariate information decomposition. We apply this framework to canonical examples from the literature and discuss the results and the various properties of the decomposition. In particular, the pointwise decomposition using specificity and ambiguity satisfies a chain rule over target variables, which provides new insights into the so-called two-bit-copy example.

Quantifying Information Modification in Developing Neural Networks via Partial Information Decomposition

“…First, it does not respect the locality of information, i.e., point-wise interpretations of this shared information are not continuous with respect to the underlying probability distribution functions [11]. Second, it suggests the presence of shared information in situations where in each realization only a single source ever holds non-zero information about a target [40]. We note that the latter is an issue for the BROJA-decomposition as well.…”

“…Debates continue on this aspect, and, in the future, it will be interesting to check the consistency of results reported here with respect to alternative decompositions, such as those presented by Finn et al [40] or Ince [42] in this issue.…”

“…This axiom suggests that if two inputs X 1 , X 2 with no mutual information between them (I(X 1 : X 2 ) = 0) are combined into an output that is simply their collection, i.e., Y = {X 1 , X 2 } then the shared part of the joint mutual information I(Y : X 1 , X 2 ) must be zero. However there are significant arguments against the inclusion of such an axiom, and in support of the presence of shared information in the two bit copy problem; see, e.g., Bertschinger et al [41], and, in this issue, by Ince [42] and Finn et al [40]. For example, there can be no measure of redundant information that simultaneously satisfies the original three PID axioms, has non-negative PI atoms, and possesses the identity property [43].…”

“…However, other measures of PID that do not rely on an optimization over the space of probability distributions (such as the one by Finn et al [40] in this special issue) may allow for the computation of a capacity for information modification-given the mechanism is known.…”

“…Several contributions to this special issue present progress on lattice-compatible distributions [40,42] and also investigate the consequences of the symmetrical, or asymmetrical treatment of information sources and targets [44] (also see the work of Olbrich et al [45] on this topic).…”