Statistical estimation of conditional Shannon entropy

Abstract

The new estimates of the conditional Shannon entropy are introduced in the framework of the model describing a discrete response variable depending on a vector of d factors having a density w.r.t. the Lebesgue measure in R d . Namely, the mixed-pair model (X, Y ) is considered where X and Y take values in R d and an arbitrary finite set, respectively. Such models include, for instance, the famous logistic regression. In contrast to the well-known Kozachenko -Leonenko estimates of unconditional entropy the proposed estimates are constructed by means of the certain spacial order statistics (or k-nearest neighbor statistics where k = k n depends on amount of observations n) and a random number of i.i.d. observations contained in the balls of specified random radii. The asymptotic unbiasedness and L 2 -consistency of the new estimates are established under simple conditions. The obtained results can be applied to the feature selection problem which is important, e.g., for medical and biological investigations.

Mutual information based feature subset selection in multivariate time series classification

“…2. Adapt to the time series case, those methods that do not assume any specific probability distribution model in the data, such as, [18,38,39,40].…”

“…The score function J in Equation 1requires the computation of the following MI terms: 1) I(T S i ; T S j ), 2) I(T S; C) and 3) I(T S i ; T S j | C). To compute them, methods that do not assume any probability distribution model in the data are considered [18,38,39,40]. All of these methods estimate the information shared between vectorvalue variables using a k-nearest neighbor strategy.…”

“…They will be adapted to measure the information shared between a TS and the associated classification variable. The methods that will be followed are the Ross method [38], Bulinski and Kozhevin method [39] and Coelho et al method [40].…”

“…Then, using ⇠ts p , ⌫p value is defined as ⌫p = { q |q 2 {1, ..., m} {p}, cq = cp, DT W (tsq, tsp)  ⇠ts p } . Then, H(C|T S) is estimated adapting the method developed by Bulinski and Kozhevin [39] as:…”

Empirical Estimation of Information Measures: A Literature Guide

“…Although the plug-in estimator of entropy is underbiased, using it in conjunction with Equation (21) yields estimates of mutual information that may be positively or negatively biased depending on the joint distribution [ 39 ]. Although largely unexplored (see [ 121 , 122 ] for recent work), the empirical estimation of mutual information between discrete X and analog Y is also of interest in a number of applications; for example, suppose that and Y is an intracellular voltage trace from visual cortex neurons.…”