On the statistical estimation of R&amp;#x00E9;nyi entropies

Liitiäinen, Elia; Lendasse, Amaury; Corona, Francesco

doi:10.1109/mlsp.2009.5306242

Cited by 6 publications

(11 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several estimators of entropy measures have been proposed for general multivariate densities f . These include consistent estimators based on entropic graphs [19,36], gap estimators [43], nearest neighbor distances [17,26,29,45], kernel density plug-in estimators [1,11,3,18,4,16], Edgeworth approximations [21], convex risk minimization [35] and orthogonal projections [25].…”

Section: Introductionmentioning

confidence: 99%

Estimation of Nonlinear Functionals of Densities With Confidence

Sricharan

Raich

Hero

2012

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

This paper introduces a class of k-nearest neighbor (k-NN) estimators called bipartite plug-in (BPI) estimators for estimating integrals of non-linear functions of a probability density, such as Shannon entropy and Rényi entropy. The density is assumed to be smooth, have bounded support, and be uniformly bounded from below on this set. Unlike previous k-NN estimators of non-linear density functionals, the proposed estimator uses data-splitting and boundary correction to achieve lower mean square error. Specifically, we assume that T i.i.d. samples Xi ∈ R d from the density are split into two pieces of cardinality M and N respectively, with M samples used for computing a k-nearest-neighbor density estimate and the remaining N samples used for empirical estimation of the integral of the density functional.By studying the statistical properties of k-NN balls, explicit rates for the bias and variance of the BPI estimator are derived in terms of the sample size, the dimension of the samples and the underlying probability distribution. Based on these results, it is possible to specify optimal choice of tuning parameters M/T , k for maximizing the rate of decrease of the mean square error (MSE). The resultant optimized BPI estimator converges faster and achieves lower mean squared error than previous k-NN entropy estimators. In addition, a central limit theorem is established for the BPI estimator that allows us to specify tight asymptotic confidence intervals.

show abstract

Section: Introductionmentioning

confidence: 99%

Estimation of Nonlinear Functionals of Densities With Confidence

Sricharan

Raich

Hero

2012

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

show abstract

“…Note that (40) provides us with an estimate of the entropy based upon the th most similar patch and, thus, varies with the choice of . Estimators of the Shannon (and Rényi) entropy based upon a combination of such estimates obtained using multiple values of have been proposed in [40]and [41]. However, such estimators require computation of distances to the -most similar patches for each patch in the cluster, a process that can be quite time consuming.…”

Section: Appendix a Entropy Estimationmentioning

confidence: 99%

“…The gradients are derived with respect to the model parameters specifically for the Gaussian channel model of the form (41) where and are the input and output of the Gaussian channel respectively, is a deterministic matrix, and is iid Gaussian noise. The authors in [26] show that achieves a slightly better estimate of the entropy than using N = 4.…”

Section: Appendix B Relation Between MI and Mmse Matrixmentioning

confidence: 99%

Practical Bounds on Image Denoising: From Estimation to Information

Chatterjee

Milanfar

2011

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

Abstract-Recently, in a previous work, we proposed a way to bound how well any given image can be denoised. The bound was computed directly from the noise-free image that was assumed to be available. In this work, we extend the formulation to the more practical case where no ground truth is available. We show that the parameters of the bounds, namely the cluster covariances and level of redundancy for patches in the image, can be estimated directly from the noise corrupted image. Further, we analyze the bounds formulation to show that these two parameters are interdependent and they, along with the bounds formulation as a whole, have a nice information-theoretic interpretation as well. The results are verified through a variety of well-motivated experiments.Index Terms-Bayesian Cramér-Rao lower bound, image clustering, image denoising, image patch model, mutual information, Rényi entropy, Shannon entropy.

show abstract

“…Because the probability P (X 1 ∈ S 2 ) turns out to admit a convenient asymptotic expression, it is possible to use Equation (8) to estimate the quantity…”

Section: Now If We Takementioning

confidence: 99%

“…The expectation (1) is also of interest in its own right and tends to appear under various scientific contexts. A significant application is found in the nonparametric estimation of Rényi entropies, where asymptotic analysis provides theoretically sound estimators [5,6,8]. Moreover, nearest neighbor distances and distributions play a major role in the understanding of nonparametric estimation in general [1,4].…”

Section: Introductionmentioning

confidence: 99%

Asymptotic Moments of Near Neighbor Distances for the Gaussian Distribution

Liitiäinen

2011

Electron. J. Probab.

View full text Add to dashboard Cite

We study the moments E[d α 1,k ] of the k-th nearest neighbor distance for independent identically distributed points in ℜ n . In the earlier literature, the case α > n has been analyzed by assuming a bounded support for the underlying density. The boundedness assumption is removed by assuming the multivariate Gaussian distribution. In this case, the nearest neighbor distances show very different behavior in comparison to earlier results. In the unbounded case, it is shown that E[d

show abstract

On the statistical estimation of Rényi entropies

Cited by 6 publications

References 13 publications

Estimation of Nonlinear Functionals of Densities With Confidence

Estimation of Nonlinear Functionals of Densities With Confidence

Practical Bounds on Image Denoising: From Estimation to Information

Asymptotic Moments of Near Neighbor Distances for the Gaussian Distribution

Contact Info

Product

Resources

About