Shannon entropy convergence results in the countable infinite case

2018

Entropy

This work addresses the problem of Shannon entropy estimation in countably infinite alphabets studying and adopting some recent convergence results of the entropy functional, which is known to be a discontinuous function in the space of probabilities in ∞-alphabets. Sufficient conditions for the convergence of the entropy are used in conjunction with some deviation inequalities (including scenarios with both finitely and infinitely supported assumptions on the target distribution). From this perspective, four plug-in histogram-based estimators are studied showing that convergence results are instrumental to derive new strong consistent estimators for the entropy. The main application of this methodology is a new data-driven partition (plug-in) estimator. This scheme uses the data to restrict the support where the distribution is estimated by finding an optimal balance between estimation and approximation errors. The proposed scheme offers a consistent (distribution-free) estimator of the entropy in ∞-alphabets and optimal rates of convergence under certain regularity conditions on the problem (finite and unknown supported assumptions and tail bounded conditions on the target distribution).

Section: Introductionsupporting

confidence: 91%

Shannon Entropy Estimation in ∞-Alphabets from Convergence Results: Studying Plug-In Estimators

2018

Entropy

“…Moving to the regime of no gain in minimax redundancy, we asume that k n ≥ u * f (n) eventually with n. We adopt results from the seminal work of Haussler and Opper [11] that offers a lower bound for the mutual information and consequently, the channel capacity that corresponds to the information radius of a family of distributions. The proof of 16 Notice that: this lower bound uses a covering argument and the adoption of the metric entropy with respect to the Hellinger distance 17 , d H : P(X ) × P(X ) −→ R + , for Λ f .…”

Section: ) Regime Of No Gain In Minimax Redundancymentioning

confidence: 99%

“…This last result is well-known for finite alphabets, however its extension to countably infinite alphabets is not straightforward due to the discontinuity of the entropy. The interested reader may be refer to[17],[18] for further details.September 23, 2018 DRAFT…”

mentioning

confidence: 99%

Universal Weak Variable-Length Source Coding on Countably Infinite Alphabets

IEEE Trans. Inform. Theory

Piantanida

2020

Self Cite

Motivated from the fact that universal source coding on countably infinite alphabets is not feasible, this work introduces the notion of "almost lossless source coding". Analog to the weak variable-length source coding problem studied by Han [3], almost lossless source coding aims at relaxing the lossless block-wise assumption to allow an average per-letter distortion that vanishes asymptotically as the blocklength goes to infinity. In this setup, we show on one hand that Shannon entropy characterizes the minimum achievable rate (similarly to the case of discrete sources) while on the other that almost lossless universal source coding becomes feasible for the family of finite-entropy stationary memoryless sources with countably infinite alphabets. Furthermore, we study a stronger notion of almost lossless universality that demands uniform convergence of the average per-letter distortion to zero, where we establish a necessary and sufficient condition for the so-called family of "envelope distributions" to achieve it. Remarkably, this condition is the same necessary and sufficient condition needed for the existence of a strongly minimax (lossless) universal source code for the family of envelope distributions.Finally, we show that an almost lossless coding scheme offers faster rate of convergence for the (minimax) redundancy compared to the well-known information radius developed for the lossless case at the expense of tolerating a non-zero distortion that vanishes to zero as the block-length grows. This shows that even when lossless universality is feasible, an almost lossless scheme can offer different regimes on the rates of convergence of the (worst case) redundancy versus the (worst case) distortion.The material in this paper was partially published in The IEEE 2016 [1] and IEEE 2017 [2] International Symposium on Information Theory (ISIT). J. F. Silva is with the Information and DecisionUniversal source coding, countably infinite alphabets, weak source coding, envelope distributions, information radius, metric entropy analysis.

“…The problem of Shannon entropy estimation is revisited from the perspective of understating the convergence properties of the entropy in the countably infinite case [1]. It is known that the entropy is not continuous function in the countable alphabet case [2], [3], [4].…”

Section: Introductionmentioning

confidence: 99%

“…17] on the limiting probability measure µ, for all the sequences {µ n : n ≥ 0} converging in reverse I-divergence to µ [5]. More recently, the work of Silva et al [1] addresses the entropy convergence studying a number of new settings that involve conditions on the limiting measure µ as well as the way the sequence {µ n : n ≥ 0} convergences to µ in the space of distributions. In this paper, we will revisit some of these new results from the perspective of their applicability to the problem of entropy estimation [9], [10], [11], [4].…”

Section: Introductionmentioning

confidence: 99%

Shannon entropy estimation from convergence results in the countable alphabet case

2013 IEEE Information Theory Workshop (ITW)

Parada

2013

Self Cite

In this paper new results for the Shannon entropy estimation and estimation of distributions, consistently in information divergence, are presented in the countable alphabet case. Sufficient conditions for the entropy convergence are adopted, including scenarios with both finitely and infinitely supported distributions. From this approach, new estimates, strong consistency results and rate of convergences are derived for various plug-in histogram-based schemes.