Mutual Information between Discrete and Continuous Data Sets

Abstract

Mutual information (MI) is a powerful method for detecting relationships between data sets. There are accurate methods for estimating MI that avoid problems with “binning” when both data sets are discrete or when both data sets are continuous. We present an accurate, non-binning MI estimator for the case of one discrete data set and one continuous data set. This case applies when measuring, for example, the relationship between base sequence and gene expression level, or the effect of a cancer drug on patient survival time. We also show how our method can be adapted to calculate the Jensen–Shannon divergence of two or more data sets.

A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula

“…Here we extend the GCMI estimate introduced above to this problem—estimating MI between a (univariate) discrete and a (potentially multivariate) continuous variable. Despite the wide applicability of such a measure, the practical issue of computing MI between discrete and continuous variables has so far received little attention [Lefakis and Fleuret, 2014; Magri et al, 2009; Ross, 2014]. One approach would be to discretize the continuous variable as described in Section 2.3 and use standard binned methods.…”

Estimation of Transfer Entropy between Discrete and Continuous Random Processes

“…We assume ergodicity of the processes when evaluating TE estimates. In all simulations, when using the nearest neighbours method, we set the number of neighbours as k = 3, as recommended in the literature [19], [24]. Besides, when using the binning method, we set the number of bins as…”

“…Inspired in the work of Kraskov et al, Ross developed a similar method to estimate mutual information for a mixed case, that is, to estimate mutual information between discrete and continuous random variables [19]. Ross estimator has been indicated as more efficient than popular binning estimator, even when binning is applied with bias correction [26].…”

“…16 indicates median time took by NN method normalized by median time took by the binning method, in 50 realizations of the processes. It is clear that the NN method is more time consuming, as mentioned in reference [19]. …”

“…The first method uses a very popular method of estimation of mutual information, which is based in adaptive partitioning of the support of the continuous variable, here called binning. The second method is based on the estimation of mutual information based on the distribution of k nearest neighbours (NN), for a mixed distribution, as proposed by Ross [19]. This paper is organized as follows: Section II establishes some notation and terminology, Section III defines TE mathematically, Section IV brings the development of the TE estimators for mixed cases, and Section V shows the results of the estimation with the TE estimators in different situations.…”