Maximum Likelihood Estimation of Functionals of Discrete Distributions

Abstract

We consider the problem of estimating functionals of discrete distributions, and focus on tight nonasymptotic analysis of the worst case squared error risk of widely used estimators. We apply concentration inequalities to analyze the random fluctuation of these estimators around their expectations, and the theory of approximation using positive linear operators to analyze the deviation of their expectations from the true functional, namely their \emph{bias}. We characterize the worst case squared error risk incurred by the Maximum Likelihood Estimator (MLE) in estimating the Shannon entropy $H(P) = \sum_{i = 1}^S -p_i \ln p_i$, and $F_\alpha(P) = \sum_{i = 1}^S p_i^\alpha,\alpha>0$, up to multiplicative constants, for any alphabet size $S\leq \infty$ and sample size $n$ for which the risk may vanish. As a corollary, for Shannon entropy estimation, we show that it is necessary and sufficient to have $n \gg S$ observations for the MLE to be consistent. In addition, we establish that it is necessary and sufficient to consider $n \gg S^{1/\alpha}$ samples for the MLE to consistently estimate $F_\alpha(P), 0<\alpha<1$. The minimax rate-optimal estimators for both problems require $S/\ln S$ and $S^{1/\alpha}/\ln S$ samples, which implies that the MLE has a strictly sub-optimal sample complexity. When $1<\alpha<3/2$, we show that the worst-case squared error rate of convergence for the MLE is $n^{-2(\alpha-1)}$ for infinite alphabet size, while the minimax squared error rate is $(n\ln n)^{-2(\alpha-1)}$. When $\alpha\geq 3/2$, the MLE achieves the minimax optimal rate $n^{-1}$ regardless of the alphabet size. As an application of the general theory, we analyze the Dirichlet prior smoothing techniques for Shannon entropy estimation. We show that no matter how we tune the parameters in the Dirichlet prior, this technique cannot achieve the minimax rates in entropy estimation.Comment: 27 pages, 1 figure, published in IEEE Transactions on Information Theor

Minimax Estimation of Functionals of Discrete Distributions

“…Our work (including the companion paper [29]) is the first to obtain the minimax rates, minimax rate-optimal estimators, and the maximum risk of MLE for estimating F α ( P ), 0 < α < 3/2, and entropy H ( P ) in the most comprehensive regime of ( S, n ) pairs. Evident from Table I is the fact that the MLE cannot achieve the minimax rates for estimation of H ( P ), and F α ( P ) when 0 < α < 3/2.…”

“…It was shown in the companion paper [29] that for n ≳ S , the maximum risk of the MLE H ( P n ) can be written as $$\underset{P\in {ℳ}_S}{\sup }{\mathbb{E}}_P{false(Hfalse(P_{n}false)-Hfalse(Pfalse)false)}^2\asymp \frac{S^2}{n^2}+\frac{{(\ln S)}^2}{n},$$ where the first term corresponds to the squared bias (defined as ${false({\mathbb{E}}_{P}Hfalse(P_{n}false)-Hfalse(Pfalse)false)}^2$, and the second term corresponds to the variance (defined as ${\mathbb{E}}_P{false(Hfalse(P_{n}false)-{\mathbb{E}}_{P}Hfalse(P_{n}false)false)}^2$). Then we can understand this mystery: when we fix S and let n → ∞, the variance dominates and we get the expression in (17).…”

“…Hence, the so-called Miller–Madow bias-corrected entropy estimator is defined by $Hfalse(P_{n}false)+\frac{S-1}{2n}$, whose maximum risk in estimating entropy was shown to be bounded from zero if n ≲ S by [29] and [52]. Thus, it still requires n ≫ S samples to consistently estimate the entropy, which is the same as MLE.…”

Justification of Logarithmic Loss via the Benefit of Side Information

“…There exist extensive literature on this subject, and we refer to [22] for a detailed review, as well as the theory and Matlab/Python implementations of entropy and mutual information estimators that achieve the minimax rates in all the regimes of sample size and support size pairs. For the recent growing literature on information measure estimation in the high-dimensional regime, we refer to [23], [24], [25], [22], [26], [27], [28], [29].…”

“…and we note that V (P (t) 0 ) in fact does not depend on t. With the help of (29), we plug R(λ, t) =R(λ, t) + λU (t) into (27), and obtaiñ…”