Estimation and Concentration of Missing Mass of Functions of Discrete Probability Distributions

Abstract

Given a positive function g from [0, 1] to the reals, the function's missing mass in a sequence of iid samples, defined as the sum of g(Pr(x)) over the missing letters x, is introduced and studied.The missing mass of a function generalizes the classical missing mass, and has several interesting connections to other related estimation problems. Minimax estimation is studied for order-α missing mass (g(p) = p α ) for both integer and non-integer values of α. Exact minimax convergence rates are obtained for the integer case. Concentration is studied for a class of functions and specific results are derived for order-α missing mass and missing Shannon entropy (g(p) = −p log p). Sub-Gaussian tail bounds with near-optimal worst-case variance factors are derived. Two new notions of concentration, named strongly sub-Gamma and filtered sub-Gaussian concentration, are introduced and shown to result in right tail bounds that are better than those obtained from sub-Gaussian concentration.Index terms-Missing mass, Good-Turing estimator, missing mass of a function, entropy, Mean squared error, minimax optimality, concentration, tail bounds, sub-Gaussian and sub-Gamma tails. I. INTRODUCTIONLet P be an arbitrary discrete distribution on an alphabet X . For x ∈ X , let p x P (x). Given a positive function g : [0, 1] → [0, ∞), we define a so-called additive function G(P ) over the distribution P as follows:

Confidence Intervals for Unobserved Events

“…where E r,n (∆) = O(1/n r−1 ). Let us compare this result to [10]. Applying Markov's inequality to (12) we obtain…”

“…In that sense, our proposed framework generalizes the missing mass problem, and introduces CIs for any r-norm of the missing probabilities, M r (X n ). Interestingly, point estimation of M r (X n ) was recently studied by Chandra and Thangaraj in quite a different context [10].…”

“…The derivation of the proposed CI utilizes several relaxations and inequalities, such as the r-norm (8) and Markov inequality (10). Therefore, it is of a reasonable concern that the obtained CI is over pessimistic.…”