Convergence properties of functional estimates for discrete distributions

Abstract

Suppose P is an arbitrary discrete distribution on a countable alphabet .Given an i.i.d. sample X 1 X n drawn from P, we consider the problem of estimating the entropy H P or some other functional F = F P of the unknown distribution P. We show that, for additive functionals satisfying mild conditions (including the cases of the mean, the entropy, and mutual information), the plug-in estimates of F are universally consistent. We also prove that, without further assumptions, no rate-of-convergence results can be obtained for any sequence of estimators. In the case of entropy estimation, under a variety of different assumptions, we get rate-of-convergence results for the plug-in estimate and for a nonparametric estimator based on match-lengths. The behavior of the variance and the expected error of the plug-in estimate is shown to be in sharp contrast to the finite-alphabet case. A number of other important examples of functionals are also treated in some detail.

Estimating the Entropy of Binary Time Series: Methodology, Some Theory and a Simulation Study

“…Moreover, there is no closed-form expression for the entropy rate of a general HMM, but, as outlined below, it is fairly easy to obtain an accurate approximation when the distribution of the HMM is known a priori, via the Shannon-McMillan-Breiman theorem (12). That is, the value of the entropy rate H = H(X) can be estimated accurately as long as it is possible to get a close approximation for the probability p n (X n 1 ) of a long random sample X n 1 .…”

“…A, necessarily incomplete, sample of the theory that has been developed can be found in [1][2][3][4][5][6][7][8][9][10][11][12][13][14] and the references therein. Examples of numerous different applications are contained in the above list, as well as in [15][16][17][18][19][20][21][22][23][24][25][26][27].…”

“…For any fixed word-length w, it is easy to see that the bias E Ĥ n,w,plug−in − 1 w H(X w 1 ) is always negative [12,13], whereas the difference between the wth-order per-symbol entropy and the entropy rate,…”

Bias Adjustment for a Nonparametric Entropy Estimator

“…Miller [2] and Basharin [3] were among the first to study nonparametric estimation of H. Since then, the topic has been investigated from a variety of directions and perspectives. Many important references can be found in [4] and [5]. In this paper, we introduce a modification of an estimator of entropy, which was first defined by Zhang in [6].…”

Coincidences and Estimation of Entropies of Random Variables with Large Cardinalities

“…Simple estimators of entropy have low variances but high biases that are difficult to calculate due to the divergence of the logarithm near zero [1]. Developments driven in part by computational biology applications have solved this problem in the moderately undersampled regime, N ∼ K and N ∼ e H [1][2][3][4][5][6][7][8][9]. Interestingly, they also resulted in the understanding that it is impossible to estimate entropy with zero bias uniformly over all distributions for a smaller N .…”