Survey on the estimation of mutual information methods as a measure of dependency versus correlation analysis

Abstract

Abstract. In this survey, we present and compare different approaches to estimate Mutual Information (MI) from data to analyse general dependencies between variables of interest in a system. We demonstrate the performance difference of MI versus correlation analysis, which is only optimal in case of linear dependencies. First, we use a piece-wise constant Bayesian methodology using a general Dirichlet prior. In this estimation method, we use a two-stage approach where we approximate the probability distribution first and then calculate the marginal and joint entropies. Here, we demonstrate the performance of this Bayesian approach versus the others for computing the dependency between different variables. We also compare these with linear correlation analysis. Finally, we apply MI and correlation analysis to the identification of the bias in the determination of the aerosol optical depth (AOD) by the satellite based Moderate Resolution Imaging Spectroradiometer (MODIS) and the ground based AErosol RObotic NETwork (AERONET). Here, we observe that the AOD measurements by these two instruments might be different for the same location. The reason of this bias is explored by quantifying the dependencies between the bias and 15 other variables including cloud cover, surface reflectivity and others.

A novel mutual information estimator to measure spike train correlations in a model thalamocortical network

“…However, a potential problem with traditional correlational analyses is that they are not optimal at estimating nonlinear dependencies (Gencaga et al 2014), which are often observed in neural networks. This drawback can be addressed by using a dependence measure known as mutual information (MI) (Cellucci et al 2005;Shannon 1948).…”

Towards identification of relevant variables in the observed aerosol optical depth bias between MODIS and AERONET observations

“…This particular product was obtained by the best relevant set of variables as determined by the highest mutual information value between the two variables. 3,4,5,6,7,8,9,10,11,12,13,15 0.756 0.924 1,2,4,5,6,8,10,11,12,13,15 0.756 0.921 1,2,4,5,7,8,10,11,12,13,15 0.755 0.923 2, 3,4,5,7,8,9,10,11,12,13,15 0.755 0.924…”

“…This relation shows the generality of the normalized correlation measure. There are several methods to estimate MI from data [15,20,5]. We applied the Variable Bin Width Histogram Approach [3,21] to compute the MI between the observed and predicted AOD.…”

A novel mutual information estimator to measure spike train correlations in a model thalamocortical network

“…Furthermore, traditional correlation analysis, such as Pearson correlation, has been shown to be optimal only in the case of linear dependencies between variables, while MI can be applied to random variables that exhibit linear as well as nonlinear dependencies (Gencaga et al 2014), which are often demonstrated in neural networks (Schöner and Kelso 1988). Comparisons of marginal entropy, meaning the average amount of information provided by, in this case, a single spike time series, have also been used to measure the relative information content of time series in the analysis of bursting and tonic firing in the thalamus (Reinagel et al 1999).…”