Statistical Inference for Rényi Entropy Functionals

Abstract

Numerous entropy-type characteristics (functionals) generalizing Rényi entropy are widely used in mathematical statistics, physics, information theory, and signal processing for characterizing uncertainty in probability distributions and distribution identification problems. We consider estimators of some entropy (integral) functionals for discrete and continuous distributions based on the number of epsilon-close vector records in the corresponding independent and identically distributed samples from two distributions. The estimators form a triangular scheme of generalized U -statistics. We show the asymptotic properties of these estimators (e.g., consistency and asymptotic normality). The results can be applied in various problems in computer science and mathematical statistics (e.g., approximate matching for random databases, record linkage, image matching).

Estimating Renyi Entropy of Discrete Distributions

“…Motivated by these and other applications, unbiased and heuristic estimators of Rényi entropy have been studied in the physics literature following [9], and asymptotically consistent and normal estimates were proposed in [45], [18]. However, no systematic study of the complexity of estimating Rényi entropy is available.…”

“…Recall from (6) that a < (1 + c α /d 2α )( √ 2 + 1) d , and therefore, a 2 = O(( √ 2 + 1) log n ) = n c 0 for some c 0 < 1. Using (18) we get…”

Estimation of entropy-type integral functionals

“…In our first theorem, we establish consistency under two different asymptotic rates of ǫ(n) with minimal density assumptions in (ii). Note that Källberg et al (2012) prove (i) under stronger density conditions (i.e. boundedness and continuity).…”

“…Henceforth we use log x to denote the natural logarithm of x. For non-negative integers k 1 , k 2 ≥ 0, k := (k 1 , k 2 ), we consider the Rényi entropy functionals (Källberg et al, 2012) q k = q k 1 ,k 2 :=…”

Model Selection Criteria Using Divergences

“…We refer to [11] for the modified versions of χ 2 and KL divergences. Some applied models using divergence and entropy measures can be found in Toma and Leoni-Aubin [27], Kallberg et al [28], Preda et al [29] and Basu et al [2], among others.…”