Measures of Entropy From Data Using Infinitely Divisible Kernels

Giraldo, Luis G. Sánchez; Rao, Murali; Prı́ncipe, José C.

doi:10.1109/tit.2014.2370058

Cited by 87 publications

(57 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, like all other information theoretic quantities, TE is defined in terms of the probability distributions of the system under study, that in practice need to be estimated from data. Probability estimation is a challenging task, and it can significantly affect the outcome of information theory analyses, including the computation of TE (Giraldo et al, 2015; Cekic et al, 2018; Timme and Lapish, 2018). Current methods that successfully estimate TE are based on a local approximation of the probability distributions from nearest neighbor distances (Kraskov et al, 2004; Lindner et al, 2011), or on symbolization schemes that then allow the probabilities to be estimated from the symbols' relative frequencies (Dimitriadis et al, 2016).…”

Section: Introductionmentioning

confidence: 99%

“…Current methods that successfully estimate TE are based on a local approximation of the probability distributions from nearest neighbor distances (Kraskov et al, 2004; Lindner et al, 2011), or on symbolization schemes that then allow the probabilities to be estimated from the symbols' relative frequencies (Dimitriadis et al, 2016). Nonetheless, obtaining TE directly from data, without the intermediate step of probability estimation, as has been achieved for other information theoretic quantities (Giraldo et al, 2015), is desirable.…”

Section: Introductionmentioning

confidence: 99%

“…Renyi's entropy is a mathematical generalization of the concept of Shannon entropy. It corresponds to a family of entropies that, because of its functional dependence on the parameter α, can emphasize either mean behavior and slowly change features in the data, or rare, uncommon events (Gao et al, 2011; Giraldo et al, 2015). This flexibility gives Renyi's entropy an advantage when it comes to analyzing data from biomedical systems (Liang et al, 2015), and has been exploited in neuroscience studies, for instance, to better characterize the randomness of EEG signals in childhood absence epilepsy (Mammone et al, 2015), and to track EEG changes associated with different anesthesia states (Liang et al, 2015).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Data-Driven Measure of Effective Connectivity Based on Renyi's α-Entropy

2019

View full text Add to dashboard Cite

Transfer entropy (TE) is a model-free effective connectivity measure based on information theory. It has been increasingly used in neuroscience because of its ability to detect unknown non-linear interactions, which makes it well suited for exploratory brain effective connectivity analyses. Like all information theoretic quantities, TE is defined regarding the probability distributions of the system under study, which in practice are unknown and must be estimated from data. Commonly used methods for TE estimation rely on a local approximation of the probability distributions from nearest neighbor distances, or on symbolization schemes that then allow the probabilities to be estimated from the symbols' relative frequencies. However, probability estimation is a challenging problem, and avoiding this intermediate step in TE computation is desirable. In this work, we propose a novel TE estimator using functionals defined on positive definite and infinitely divisible kernels matrices that approximate Renyi's entropy measures of order α. Our data-driven approach estimates TE directly from data, sidestepping the need for probability distribution estimation. Also, the proposed estimator encompasses the well-known definition of TE as a sum of Shannon entropies in the limiting case when α → 1. We tested our proposal on a simulation framework consisting of two linear models, based on autoregressive approaches and a linear coupling function, respectively, and on the public electroencephalogram (EEG) database BCI Competition IV, obtained under a motor imagery paradigm. For the synthetic data, the proposed kernel-based TE estimation method satisfactorily identifies the causal interactions present in the data. Also, it displays robustness to varying noise levels and data sizes, and to the presence of multiple interaction delays in the same connected network. Obtained results for the motor imagery task show that our approach codes discriminant spatiotemporal patterns for the left and right-hand motor imagination tasks, with classification performances that compare favorably to the state-of-the-art.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Data-Driven Measure of Effective Connectivity Based on Renyi's α-Entropy

2019

View full text Add to dashboard Cite

show abstract

“…where A(C, P) = C,P F C F P F is exactly the kernel alignment cost function [12,54]. Note that the distance in Equation 6 can be implemented also with more advanced differentiable measures of (dis)similarity between positive-definite matrices, such as divergence and mutual information [14,32]. However, these options are not explored in this paper and are left for future research.…”

Section: Deep Kernelized Autoencodersmentioning

confidence: 99%

The deep kernelized autoencoder

Kampffmeyer¹,

Løkse²,

Bianchi³

et al. 2018

Applied Soft Computing

View full text Add to dashboard Cite

Autoencoders learn data representations (codes) in such a way that the input is reproduced at the output of the network. However, it is not always clear what kind of properties of the input data need to be captured by the codes. Kernel machines have experienced great success by operating via inner-products in a theoretically well-defined reproducing kernel Hilbert space, hence capturing topological properties of input data. In this paper, we enhance the autoencoder's ability to learn effective data representations by aligning inner products between codes with respect to a kernel matrix. By doing so, the proposed kernelized autoencoder allows learning similarity-preserving embeddings of input data, where the notion of similarity is explicitly controlled by the user and encoded in a positive semi-definite kernel matrix. Experiments are performed for evaluating both reconstruction and kernel alignment performance in classification tasks and visualization of high-dimensional data. Additionally, we show that our method is capable to emulate kernel principal component analysis on a denoising task, obtaining competitive results at a much lower computational cost.

show abstract

“…is a test statistic that is proportional to Rényi's quadratic entropy [13] under the model defined by P . Thus, for succinctness, we refer to Eq.…”

Section: Kernel-based Quadratic Entropymentioning

confidence: 99%

Projentropy: Using entropy to optimize spatial projections

Brockmeier

Santanna

Giraldo

et al. 2014

2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Methods for hypothesis testing on zero-mean vector-valued signals often rely on a Gaussian assumption, where the second-order statistics of the observed sample are sufficient statistics of the conditional distribution. This yields fast and simple tests, but by using information-theoretic statistics one can relax the Gaussian assumption. We propose using Rényi's quadratic entropy as an alternative to the covariance and show how a linear projection can be optimized to maximize the difference between the conditional entropies. In addition, if the observed sample is actually a window of a multivariate time-series, then the temporal structure can be exploited using the generalized auto-correlation function, correntropy, of the projected sample. This both reduces the computational complexity and increases the performance. These tests can be applied for decoding the brain state from electroencephalogram (EEG) recordings. Preliminary results are demonstrated on a brain-computer interface competition dataset. On unfiltered signals, the projections optimized with the entropy-based statistic perform better than those of common spatial pattern (CSP) algorithm in terms of classification performance.

show abstract

Measures of Entropy From Data Using Infinitely Divisible Kernels

Cited by 87 publications

References 17 publications

A Data-Driven Measure of Effective Connectivity Based on Renyi's α-Entropy

A Data-Driven Measure of Effective Connectivity Based on Renyi's α-Entropy

The deep kernelized autoencoder

Projentropy: Using entropy to optimize spatial projections

Contact Info

Product

Resources

About