Contribution to Transfer Entropy Estimation via the k-Nearest-Neighbors Approach

Zhu, Jianping; Bellanger, Jean-Jacques; Hu, Shu; Jeannès, Régine Le Bouquin

doi:10.3390/e17064173

Cited by 27 publications

(19 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, with TE-kraskov, a very slow convergence of information exchange with increasing time series length is noted. This behavior was reported in the study by [ 49 ]. In summary, for this system, IF-linear is able to detect the asymmetry in the information exchange while TE-linear fails.…”

Section: Resultssupporting

confidence: 84%

Quantification of Information Exchange in Idealized and Climate System Applications

Pothapakula

Primo

Ahrens

2019

Entropy

View full text Add to dashboard Cite

Often in climate system studies, linear and symmetric statistical measures are applied to quantify interactions among subsystems or variables. However, they do not allow identification of the driving and responding subsystems. Therefore, in this study, we aimed to apply asymmetric measures from information theory: the axiomatically proposed transfer entropy and the first principle-based information flow to detect and quantify climate interactions. As their estimations are challenging, we initially tested nonparametric estimators like transfer entropy (TE)-binning, TE-kernel, and TE k-nearest neighbor and parametric estimators like TE-linear and information flow (IF)-linear with idealized two-dimensional test cases along with their sensitivity on sample size. Thereafter, we experimentally applied these methods to the Lorenz-96 model and to two real climate phenomena, i.e., (1) the Indo-Pacific Ocean coupling and (2) North Atlantic Oscillation (NAO)–European air temperature coupling. As expected, the linear estimators work for linear systems but fail for strongly nonlinear systems. The TE-kernel and TE k-nearest neighbor estimators are reliable for linear and nonlinear systems. Nevertheless, the nonparametric methods are sensitive to parameter selection and sample size. Thus, this work proposes a composite use of the TE-kernel and TE k-nearest neighbor estimators along with parameter testing for consistent results. The revealed information exchange in Lorenz-96 is dominated by the slow subsystem component. For real climate phenomena, expected bidirectional information exchange between the Indian and Pacific SSTs was detected. Furthermore, expected information exchange from NAO to European air temperature was detected, but also unexpected reversal information exchange. The latter might hint to a hidden process driving both the NAO and European temperatures. Hence, the limitations, availability of time series length and the system at hand must be taken into account before drawing any conclusions from TE and IF-linear estimations.

show abstract

Section: Resultssupporting

confidence: 84%

Quantification of Information Exchange in Idealized and Climate System Applications

Pothapakula

Primo

Ahrens

2019

Entropy

View full text Add to dashboard Cite

show abstract

“…Moreover, it is sensitive to the size of bins used. Other nonparametric entropy estimation methods have been also used for computing the TE [ 21 , 22 , 23 ]: kernel density estimation methods, nearest-neighbor, Parzen, neural networks, etc.…”

Section: Background: Transfer Information Entropymentioning

confidence: 99%

Learning in Feedforward Neural Networks Accelerated by Transfer Entropy

Moldovan

Caţaron

Andonie

2020

Entropy

View full text Add to dashboard Cite

Current neural networks architectures are many times harder to train because of the increasing size and complexity of the used datasets. Our objective is to design more efficient training algorithms utilizing causal relationships inferred from neural networks. The transfer entropy (TE) was initially introduced as an information transfer measure used to quantify the statistical coherence between events (time series). Later, it was related to causality, even if they are not the same. There are only few papers reporting applications of causality or TE in neural networks. Our contribution is an information-theoretical method for analyzing information transfer between the nodes of feedforward neural networks. The information transfer is measured by the TE of feedback neural connections. Intuitively, TE measures the relevance of a connection in the network and the feedback amplifies this connection. We introduce a backpropagation type training algorithm that uses TE feedback connections to improve its performance.

show abstract

“…(1) by recognizing it as the expectation of the logarithm of the Radon-Nikodym derivative of a given conditional probability measure with respect to a distinct, but equivalent, 2 , conditional probability measure (as observed in [36]). We point out that the Radon-Nikodym derivative serves as the density of a measure with respect to another and can function as a generalized Jacobian, changing between those measures under an integral, analogously to a normal derivative.…”

Section: Transfer Entropy In Continuous Timementioning

confidence: 99%

Transfer entropy in continuous time, with applications to jump and neural spiking processes

2017

View full text Add to dashboard Cite

Transfer entropy has been used to quantify the directed flow of information between source and target variables in many complex systems. While transfer entropy was originally formulated in discrete time, in this paper we provide a framework for considering transfer entropy in continuous time systems, based on Radon-Nikodym derivatives between measures of complete path realizations. To describe the information dynamics of individual path realizations, we introduce the pathwise transfer entropy, the expectation of which is the transfer entropy accumulated over a finite time interval. We demonstrate that this formalism permits an instantaneous transfer entropy rate. These properties are analogous to the behavior of physical quantities defined along paths such as work and heat. We use this approach to produce an explicit form for the transfer entropy for pure jump processes, and highlight the simplified form in the specific case of point processes (frequently used in neuroscience to model neural spike trains). Finally, we present two synthetic spiking neuron model examples to exhibit the pertinent features of our formalism, namely, that the information flow for point processes consists of discontinuous jump contributions (at spikes in the target) interrupting a continuously varying contribution (relating to waiting times between target spikes). Numerical schemes based on our formalism promise significant benefits over existing strategies based on discrete time formalisms.

show abstract

Contribution to Transfer Entropy Estimation via the k-Nearest-Neighbors Approach

Cited by 27 publications

References 41 publications

Quantification of Information Exchange in Idealized and Climate System Applications

Quantification of Information Exchange in Idealized and Climate System Applications

Learning in Feedforward Neural Networks Accelerated by Transfer Entropy

Transfer entropy in continuous time, with applications to jump and neural spiking processes

Contact Info

Product

Resources

About