Network Information and Connected Correlations

Abstract

Entropy and information provide natural measures of correlation among elements in a network. We construct here the information theoretic analog of connected correlation functions: irreducible N -point correlation is measured by a decrease in entropy for the joint distribution of N variables relative to the maximum entropy allowed by all the observed N − 1 variable distributions. We calculate the "connected information" terms for several examples, and show that it also enables the decomposition of the information that is carried by a population of elements about an outside source. Keywords: entropy, information, multi-information, redundancy, synergy, correlation, network In statistical physics and field theory, the nature of order in a system is characterized by correlation functions. These ideas are especially powerful because there is a direct relation between the correlation functions and experimental observables such as scattering cross sections and susceptibilities. As we move toward the analysis of more complex systems, such as the interactions among genes or neurons in a network, it is not obvious how to construct correlation functions which capture the underlying order. On the other hand it is possible to observe directly the activity of many single neurons in a network or the expression levels of many genes, and hence real experiments in these systems are more like Monte Carlo simulations, sampling the distribution of network states.Shannon proved that, given a probability distribution over a set of variables, entropy is the unique measure of what can be learned by observing these variables, given certain simple and plausible criteria (continuity, monotonicity and additivity) [1]. By the same arguments, mutual information arises as the unique measure of the interdependence of two variables, or two sets of variables. Defining information theoretic analogs of higher order correlations has proved to be more difficult [2,3,4,5,6,7,8,9, 10]. When we compute N -point correlation functions in statistical physics and field theory, we are careful to isolate the connected correlations, which are the components of the N -point correlation that cannot be factored into correlations among groups of fewer than N observables. We propose here an analogous measure of "connected information" which generalizes precisely our intuition about connectedness and interactions from field theory; a closely related discussion for quantum information has been given recently [11].Consider N variables {x i }, i = 1, 2, ..., N , drawn from the joint probability distribution P ({x i }); this has an entropy [12]. S({x(The fact that N variables are correlated means that the entropy S({x i }) is smaller than the sum of the entropies for each variable individually,The total difference in entropy between the interacting variables and the variables taken independently can be written as [2,3]

Local statistics in natural scenes predict the saliency of synthetic textures

“…Each of these distributions has its associated entropy, S ðνÞ G ¼ −∑~σP ðνÞ G ð~σÞ log 2 P ðνÞ G ð~σÞ. Following (27), ν-th order correlations within the glider carry I ðνÞ G ¼ S ðν−1Þ G − S ðνÞ G bits of information about local texture. These information-theoretic quantities measure order in a texture that arises from correlations that involve exactly ν pixels.…”

“…Because pixels are not independent, the entropy S G will in general be less than four bits; we can write S G ¼ Q − ∑ Q ν¼1 I ðνÞ G where Q ¼ 4 is the number of binary pixels in the glider, and I ðνÞ G measures the bits of entropy reduction caused by luminance bias (ν ¼ 1), and by pair (ν ¼ 2), triplet (ν ¼ 3), and quadruplet (ν ¼ 4) correlations (27).…”

GHZ extraction yield for multipartite stabilizer states

“…A similar measure has been introduced also for multipartite probability distributions in Ref. 18. It will allow us to show that ⌬͑⌿͒ is equal to the number of GHZ states extractable from ͉⌿͘ by arbitrary local unitaries.…”

Pairwise network information and nonlinear correlations

“…The fraction of information retained by describing the system with a given measure, as opposed to the true joint entropy, is then 0 ≤ I m /I N ≤ 1. If the used measure is the bivariate probability distribution, we call I m the pairwise network information, or the second order connected information as defined in [16]. This is approximated linearly if the measure used is the cross-correlation, and nonlinearly if the measure used is the mutual information.…”

“…The corresponding framework was first laid out in [16] and later applied in [17], where they assessed the rationale of only looking at the pairwise correlations between neurons. They examined how well the maximum entropy distribution, consistent with all the pairwise correlations described the system.…”