Does dirichlet prior smoothing solve the Shannon entropy estimation problem?

Abstract

Abstract-We consider the problem of estimating functionals of discrete distributions, and focus on tight (up to universal multiplicative constants for each specific functional) nonasymptotic analysis of the worst case squared error risk of widely used estimators. We apply concentration inequalities to analyze the random fluctuation of these estimators around their expectations, and the theory of approximation using positive linear operators to analyze the deviation of their expectations from the true functional, namely their bias.We explicitly characterize the worst case squared error risk incurred by the Maximum Likelihood Estimator (MLE) in estimating the Shannon entropy H(P ) = S i=1 −pi ln pi, and the power sum Fα(P ) = S i=1 p α i , α > 0, up to universal multiplicative constants for each fixed functional, for any alphabet size S ≤ ∞ and sample size n for which the risk may vanish. As a corollary, for Shannon entropy estimation, we show that it is necessary and sufficient to have n ≫ S observations for the MLE to be consistent. In addition, we establish that it is necessary and sufficient to consider n ≫ S 1/α samples for the MLE to consistently estimate Fα(P ), 0 < α < 1. The minimax rate-optimal estimators for both problems require S/ ln S and S 1/α / ln S samples, which implies that the MLE has a strictly sub-optimal sample complexity. When 1 < α < 3/2, we show that the worst-case squared error rate of convergence for the MLE is n −2(α−1) for infinite alphabet size, while the minimax squared error rate is (n ln n) −2(α−1) . When α ≥ 3/2, the MLE achieves the minimax optimal rate n −1 regardless of the alphabet size.As an application of the general theory, we analyze the Dirichlet prior smoothing techniques for Shannon entropy estimation. In this context, one approach is to plug-in the Dirichlet prior smoothed distribution into the entropy functional, while the other one is to calculate the Bayes estimator for entropy under the Dirichlet prior for squared error, which is the conditional expectation. We show that in general such estimators do not improve over the maximum likelihood estimator. No matter how we tune the parameters in the Dirichlet prior, this approach cannot achieve the minimax rates in entropy estimation. The performance of the minimax rate-optimal estimator with n samples is essentially at least as good as that of Dirichlet smoothed entropy estimators with n ln n samples.

Minimax Estimation of Functionals of Discrete Distributions

“…Wolpert and Wolf [56] gives an explicit expression of this estimator: $${\widehat{H}}^{\text{Bayes}}=true\sum _{i=1}^S-\frac{n{\widehat{p}}_i+a}{n+Sa}·false({\psi }_{0}false(n+Sa+1false)-{\psi }_{0}false(n{\widehat{p}}_i+a+1false)false).$$It has been shown in [58] that this approach cannot achieve the minimax rates. We feed the algorithm with true S and set $a=\sqrt{n}/S$ in our experiments.…”

“…Bearing this in mind, Figure 5 suggests that the following three estimators out of 12, namely our estimator, the estimator by Valiant and Valiant [25] and the best upper bound (BUB) estimator, achieve the minimax rate. Some other estimators have already been shown not to achieve the optimal sample complexity, e.g., the Miller-Madow bias-corrected MLE [29], [52], the jackknifed estimator [52], and the Dirichlet-smoothed plug-in estimator as well as the Bayes estimator under Dirichlet prior [58]. We remark that the shrinkage estimator only improves the MLE when the distribution is near-uniform, and this estimator performs poorly under the Zipf distribution (verified by our experiments), thus it attains neither the optimal minimum sample complexity nor the minimax rate.…”

“…We thank Yihong Wu for insightful discussions, in particular, regarding a non-rigorous step in the proof of Lemma 16 in a previous version of the manuscript. They thank Maya Gupta for inspiring discussions, in particular for raising the question of whether Dirichlet prior smoothed plug-in entropy estimation can achieve the minimax rates, whose answer was shown to be negative in [58]. Finally, they thank Dmitri Sergeevich Pavlichin for translating articles from Bernstein’s collected works [189], whose English versions are unavailable.…”

Importance of Small Probability Events in Big Data: Information Measures, Applications, and Challenges

“…Besides, the estimator based on the best polynomial approximation also has the same performance [123]. Moreover, an inferior estimator is constructed by use of Dirichlet prior smoothing, which is similar to MLE but not as good as the above two [124]. In addition, an ensemble of plug-in estimators with weights is proposed to protect the results of estimation from decaying with the increase of sample dimension [125].…”

Minimax Optimal Additive Functional Estimation with Discrete Distribution: Slow Divergence Speed Case

“…- [24] still inconsistent if k is larger than n. The estimators based on Bayesian approaches in [16]- [18] are also inconsistent for k n [19]. Paninski [20] firstly revealed existence of a consistent estimator even if the alphabet size k is larger than linear order of the sample size n. However, they did not provide a concrete form of the consistent estimator.…”