A Recipe for the Estimation of Information Flow in a Dynamical System

Abstract

Information-theoretic quantities, such as entropy and mutual information (MI), can be used to quantify the amount of information needed to describe a dataset or the information shared between two datasets. In the case of a dynamical system, the behavior of the relevant variables can be tightly coupled, such that information about one variable at a given instance in time may provide information about other variables at later instances in time. This is often viewed as a flow of information, and tracking such a flow can reveal relationships among the system variables. Since the MI is a symmetric quantity; an asymmetric quantity, called Transfer Entropy (TE), has been proposed to estimate the directionality of the coupling. However, accurate estimation of entropy-based measures is notoriously difficult. Every method has its own free tuning parameter(s) and there is no consensus on an optimal way of estimating the TE from a dataset. We propose a new methodology to estimate TE and apply a set of methods together as an accuracy cross-check to provide a reliable mathematical tool for any given data set. We demonstrate both the variability in TE estimation across techniques as well as the benefits of the proposed methodology to reliably estimate the directionality of coupling among variables.

Survey on the estimation of mutual information methods as a measure of dependency versus correlation analysis

“…p(x, y) = p(x)p(y) . As (3) is not bounded from above, normalized versions have been proposed in the literature [8,9]. For continuous variables, a normalized MI is given by…”

“…$ as we can also find the optimal number of bins by a similar algebra and estimate it as follows [8]: (10) where the log posterior pdf of the number of bins is given as follows:…”

“…Here, we extend the piece-wise constant Bayesian methodology in [7] using a general Dirichlet prior in a similar way that is utilized to estimate different information-theoretic quantities like the transfer entropy in [8]. In this estimation method, we use a two-stage approach where we approximate the probability distribution first and then calculate the marginal and joint entropies to be substituted in the MI formula estimation [7,8]. Here, we demonstrate the performance of this Bayesian approach versus the others for computing the dependency between different variables.…”

Testing the Beta-Lognormal Model in Amazonian Rainfall Fields Using the Generalized Space q-Entropy

“…We use a set of parametric and non-parametric bin-counting methods, such as those introduced by Sturges [69], Dixon and Kronmal [70], Scott [71], Freedman and Diaconis [72], Knuth [73,74], Shimazaki and Shinomoto [75,76], and a recent method for estimating entropy in hydrologic data proposed by Gong et al [67]. In work, we use common techniques reported in the literature, although there are other methods [77].…”

Symbolic Information Flow Measurement (SIFM): A Software for Measurement of Information Flow Using Symbolic Analysis

“…Computing TE is a challenging problem due to its computational complexity. Therefore, different numerical recipes have been suggested [5]. TE has already been used for time series analysis in different fields, such as clinical electroencephalography [4, 6, 7], financial data [8], and biophysics [2].…”