On convergence properties of Shannon entropy

Abstract

Convergence properties of Shannon Entropy are studied. In the differential setting, it is shown that weak convergence of probability measures, or convergence in distribution, is not enough for convergence of the associated differential entropies. A general result for the desired differential entropy convergence is provided, taking into account both compactly and uncompactly supported densities. Convergence of differential entropy is also characterized in terms of the Kullback-Liebler discriminant for densities with fairly general supports, and it is shown that convergence in variation of probability measures guarantees such convergence under an appropriate boundedness condition on the densities involved. Results for the discrete setting are also provided, allowing for infinitely supported probability measures, by taking advantage of the equivalence between weak convergence and convergence in variation in this setting.

Shannon entropy estimation from convergence results in the countable alphabet case

“…In this context Piera et al [8] introduced the following result. Interpreting the result, to obtain the convergence of the entropy without imposing a finite support assumption on the limiting measure µ, a uniform bounding condition (UBC), µ-almost surely, was added in (12).…”

“…Then the reverse I-divergence and the entropy difference are bounded by the total variation, by (8) and (9). Note that this bound is distribution dependent as it depends explicitly on m µ (M µ ) in (7).…”

“…It is known for countable alphabets that convergence in variation is equivalent to the weak convergence and the pointwise convergence [8], which is denoted by µ n ⇒ µ. Finally, we achieve convergence in reverse and direct I-divergence if lim n→∞ D(µ n ||µ) = 0 and lim n→∞ D(µ||µ n ) = 0, respectively [5].…”

“…This discontinuity rises the problem of finding conditions under which convergence of the entropy can be obtained [8]. On this topic we can mention some recent contributions.…”

Entropy and density approximation from Laplace transforms

“…See the work of Piera and Parada [18] and of Silva and Parada [19] for interesting results and further references to this problem.…”