Decomposing and Tracing Mutual Information by Quantifying Reachable Decision Regions

Abstract

The idea of a partial information decomposition (PID) gained significant attention for attributing the components of mutual information from multiple variables about a target to being unique, redundant/shared or synergetic. Since the original measure for this analysis was criticized, several alternatives have been proposed but have failed to satisfy the desired axioms, an inclusion–exclusion principle or have resulted in negative partial information components. For constructing a measure, we interpret the achievable type I/II error pairs for predicting each state of a target variable (reachable decision regions) as notions of pointwise uncertainty. For this representation of uncertainty, we construct a distributive lattice with mutual information as consistent valuation and obtain an algebra for the constructed measure. The resulting definition satisfies the original axioms, an inclusion–exclusion principle and provides a non-negative decomposition for an arbitrary number of variables. We demonstrate practical applications of this approach by tracing the flow of information through Markov chains. This can be used to model and analyze the flow of information in communication networks or data processing systems.

Non-Negative Decomposition of Multivariate Information: From Minimum to Blackwell-Specific Information

“…In a recent work [20], we presented a decomposition of mutual information on the redundancy lattice (Figure 1b). This work aims to simplify, generalize, and extend these ideas to make the following contributions to the area of partial information decompositions:…”

“…Therefore, a channel zonogon Z(κ) provides the set of all achievable (TPR,FPR)-pairs for a given channel κ [20,22]. This can also be seen from Equation (3), where the unit cube a ∈ [0, 1] |S| represents all possible first columns of the decision strategy λ.…”

“…Definition 20 ). Rényi-information is defined equivalent to f -information as shown in Equation (20) and corresponds to an invertible transformation of Hellingerinformation (I H a ). I R a (S; T) := R a P (S;T)…”

“…As discussed before [20], it is possible to further split redundancy into two components for extracting the pointwise meet under the Blackwell order (zonogon intersection, first component). The second component of redundancy as defined above contains decision regions that are part of the convex hull, but not the individual channel zonogons (discussed as shared information in [20]). By combining Equation (43) and Lemma A7, we obtain that both components of this split for redundancy are non-negative.…”

“…Remark 12. The relation between the redundancy and synergy lattice can be used for the definition of a bi-valuation [19] in calculations as discussed in [20]. This is also possible for Rényi-information at transformed operators.…”