Shannon Entropy Estimation in ∞-Alphabets from Convergence Results: Studying Plug-In Estimators

Abstract

This work addresses the problem of Shannon entropy estimation in countably infinite alphabets studying and adopting some recent convergence results of the entropy functional, which is known to be a discontinuous function in the space of probabilities in ∞-alphabets. Sufficient conditions for the convergence of the entropy are used in conjunction with some deviation inequalities (including scenarios with both finitely and infinitely supported assumptions on the target distribution). From this perspective, four plug-in histogram-based estimators are studied showing that convergence results are instrumental to derive new strong consistent estimators for the entropy. The main application of this methodology is a new data-driven partition (plug-in) estimator. This scheme uses the data to restrict the support where the distribution is estimated by finding an optimal balance between estimation and approximation errors. The proposed scheme offers a consistent (distribution-free) estimator of the entropy in ∞-alphabets and optimal rates of convergence under certain regularity conditions on the problem (finite and unknown supported assumptions and tail bounded conditions on the target distribution).

On Universal D-Semifaithful Coding for Memoryless Sources With Infinite Alphabets

“…On the constructive side, constraining the problem to a family of distributions with specific power tail-bounded conditions, Antos et al [36,Theorem 7] presented a finite length expression for the rate of convergence of the estimation error of the classical plug-in estimator. Similar results -strong consistency distribution-free in H(X) and (almost sure) rate of convergence for the estimation error under some tail bounded conditions -have been obtained for a datadriven partition scheme in [37]. 15 .…”

“…B, (39) 15 As a side comment, the mentioned tail bounded conditions used in [36], [37] to obtain rate of convergence for entropy estimation are stronger than the condition used in our work to define the envelope families (Definition 4) -used in our achievability results in Theorems 2 and 3 where ⇡ n (A n j ) ⌘ B 2 ⇡ n , A n j \ B 6 = ; . At this point, we can show that C(A n j ) < 1, 8j 2 J .…”

“…Given this capability, it is interesting to mention some results for entropy estimation and distribution estimation for infinite alphabets. We focus on the lossless coding setting (d = 0) as this regime offers a connection with the problems of entropy estimation and distribution estimation in information divergence [4], [22], [36], [37].…”

Staircase patterns in words: Subsequences, subwords, and separation number

“…We thus only prove the former in Section 2.3 and omit the proof of the latter. We remark that Theorem 1.3 implies that H (lw) k (n) converges, as n → ∞, to the entropy of the limiting distribution of the sequence I k (n) (see, for instance, Proposition 1 in [37] -some care is required because, in general, the entropy is not a continuous function of distributions in infinite spaces, even in a discrete setting [11,37]).…”

Dimension-Free Empirical Entropy Estimation

“…Even though H(•) is not continuous on ∆ N , the plug-in estimate H( μn ) converges to H(µ) almost surely and in L 2 [Antos and Kontoyiannis, 2001a]. Silva [2018] studied a variety of restrictions on distributions over infinite alphabets to derive strong consistency results and rates of convergence.…”