Minimax estimation of discrete distributions

Abstract

We refine the general methodology in [1] for the construction and analysis of essentially minimax estimators for a wide class of functionals of finite dimensional parameters, and elaborate on the case of discrete distributions with support size S comparable with the number of observations n. Specifically, we determine the "smooth" and "non-smooth" regimes based on the confidence set and the smoothness of the functional. In the "non-smooth" regime, we apply an unbiased estimator for a suitable polynomial approximation of the functional. In the "smooth" regime, we construct a general version of the bias-corrected Maximum Likelihood Estimator (MLE) based on Taylor expansion.We apply the general methodology to the problem of estimating the KL divergence between two discrete probability measures P and Q from empirical data in a non-asymptotic and possibly large alphabet setting. We construct minimax rate-optimal estimators for D(P Q) when the likelihood ratio is upper bounded by a constant which may depend on the support size, and show that the performance of the optimal estimator with n samples is essentially that of the MLE with n ln n samples. Our estimator is adaptive in the sense that it does not require the knowledge of the support size nor the upper bound on the likelihood ratio. We show that the general methodology results in minimax rate-optimal estimators for other divergences as well, such as the Hellinger distance and the χ 2 -divergence. Our approach refines the Approximation methodology recently developed for the construction of near minimax estimators of functionals of high-dimensional parameters, such as entropy, Rényi entropy, mutual information and 1 distance in large alphabet settings, and shows that the effective sample size enlargement phenomenon holds significantly more widely than previously established.

The Expected Missing Mass under an Entropy Constraint

“…Another possible restriction that makes the problem interesting is that the entropy of p is bounded above by some given value. A similar restriction can be found in [10] in the context of discrete distribution estimation under 1 loss. In this work, we study the possibility of providing tight bounds on EU t under the restriction of some bound on the entropy.…”

Minimax Estimation of Functionals of Discrete Distributions

“…The reasons are twofold: the supersymbols Y i are no longer independent, and it satisfies the asymptotic equipartition property (AEP) by the Shannon-McMillan-Breiman theorem [178]. To see the role played by ergodicity, it has been shown in [179] that minimax estimation of discrete distributions with support size S under ℓ 1 loss requires n ≫ S samples, which turns out to be S D ≪ n , or equivalently, $D\ll \frac{\ln n}{\ln S}$, in the D -tuple distribution estimation in stochastic processes. However, [180] showed that it suffices to choose the memory length $D\le false(1-\epsilon false)\frac{\ln n}{H}$ for any ε > 0 in a stationary ergodic stochastic process, where H is its entropy rate, to guarantee that the empirical joint distribution of D -tuple converges to the true joint distribution under ℓ 1 loss.…”

“…samples of a random vector X = ( X 1 , X 2 , …, X d ), where $X_i\in \mathrm{scriptZ}$, $false|\mathrm{scriptZ}false|<\infty$, we are interested in estimating the joint distribution of X . It was shown [179] that one needs to take $n\gg false|\mathrm{scriptZ}{|}^d$ samples to consistently estimate the joint distribution [40], which blows up quickly with growing d . Practically, it is convenient and necessary to impose some structure on the joint distribution P X to reduce the required sample complexity.…”

Does dirichlet prior smoothing solve the Shannon entropy estimation problem?

“…, δ S (X)) is the minimax estimator for P under squared loss [46,Example 5.4.5]. However, it is no longer minimax under other loss functions such as ℓ 1 loss, which was investigated in [47].…”

“…Wu and Yang [21] independently applied the best polynomial approximation idea to entropy estimation and obtained the minimax rates. However, their estimator requires the knowledge of the alphabet size S. The approximation ideas proved to be very fruitful in Acharya et al [22], Wu and Yang [23], Han, Jiao, and Weissman [24], Jiao, Han, and Weissman [25], Bu et al [26], Orlitsky, Suresh, and Wu [27], Wu and Yang [28].…”