Rediscovery of Good–Turing Estimators via Bayesian Nonparametrics

Abstract

Summary. The problem of estimating discovery probabilities originated in the context of statistical ecology, and in recent years it has become popular due to its frequent appearance in challenging applications arising in genetics, bioinformatics, linguistics, designs of experiments, machine learning, etc. A full range of statistical approaches, parametric and nonparametric as well as frequentist and Bayesian, has been proposed for estimating discovery probabilities. In this article, we investigate the relationships between the celebrated Good-Turing approach, which is a frequentist nonparametric approach developed in the 1940s, and a Bayesian nonparametric approach recently introduced in the literature. Specifically, under the assumption of a two parameter Poisson-Dirichlet prior, we show that Bayesian nonparametric estimators of discovery probabilities are asymptotically equivalent, for a large sample size, to suitably smoothed Good-Turing estimators. As a by-product of this result, we introduce and investigate a methodology for deriving exact and asymptotic credible intervals to be associated with the Bayesian nonparametric estimators of discovery probabilities. The proposed methodology is illustrated through a comprehensive simulation study and the analysis of Expressed Sequence Tags data generated by sequencing a benchmark complementary DNA library.

Bayesian nonparametric inference for discovery probabilities: credible intervals and large sample asymptotics

“…Under the two-parameter PD prior, Favaro et al (2016) established a large n asymptotic relationship betweenD n (l) andĎ n (l). Due to the irregular behaviour of the m l,m 's, the peculiar dependency on m l+1,n makesĎ n (l) a sensible estimator only if l is sufficiently small with respect to n. See for instance Good (1953) and Sampson (2001) for examples of absurd estimates determined byĎ n (l).…”

“…See Chapter 7 in Sampson (2001) and references therein for a comprehensive account on smoothing techniques forĎ n (l). According to Theorem 1 in Favaro et al (2016), as n becomes large,D n (l) is asymptotically equivalent toĎ n (l; S PD ), where S PD denotes a smoothing rule such that…”

“…While the smoothing approach was introduced as an ad hoc tool for post processing the irregular m l,n 's in order to improve the performance ofĎ n (l), Theorem 1 in Favaro et al (2016) shows that, for a large sample size n, a similar smoothing mechanism underlies the Bayesian nonparametric framework (2) with a two-parameter PD prior. Interestingly, the smoothing rule S PD has been proved to be a generalization of the Poisson smoothing rule discussed in Good (1953) and Engen (1978).…”

“…The problem of associating a measure of uncertainty to Bayesian nonparametric estimators for discovery probabilities was first addressed in Lijoi et al (2007) where estimates of the probability of observing a new species are endowed with highest posterior density intervals. Favaro et al (2016) derive asymptotic posterior credible intervals covering also the case of species already observed with a given frequency. These contributions ultimately rely on the presence of an additional unobserved sample.…”

“…These contributions ultimately rely on the presence of an additional unobserved sample. While the approach of Lijoi et al (2007) cannot be used to associate a measure of uncertainty toD n (0), where such additional sample is not considered, the approach of Favaro et al (2016) could be taken to derive approximate credible intervals forD n (l), l = 0, 1, . .…”