Bayesian Nonparametric Inference for "Species-sampling" Problems

Abstract

Species-sampling" problems (SSPs) refer to a broad class of statistical problems that, given an observable sample from an unknown population of individuals belonging to some species, call for estimating features of the unknown species composition of additional unobservable samples from the same population. Among SSPs, the problems of estimating coverage probabilities, the number of unseen species and coverages of prevalences have emerged over the past three decades for being the objects of numerous studies, both in methods and applications, mostly within the field of biological sciences but also in machine learning, electrical engineering, theoretical computer science and information theory. In this paper, we present an overview of Bayesian nonparametric (BNP) inference for such three SSPs under the popular Pitman-Yor process (PYP) prior: i) we introduce each SSP in the classical (frequentist) nonparametric framework, and review its posterior analyses in the BNP framework; ii) we improve on computation and interpretability of existing posterior distributions, typically expressed through complicated combinatorial numbers, by establishing novel posterior representations in terms of simple compound Binomial and Hypergeometric distributions. The critical question of estimating the discount and scale parameters of the PYP prior is also considered and investigated, establishing a general property of Bayesian consistency with respect to the hierarchical Bayes and empirical Bayes approaches, that is: the discount parameter can be always estimated consistently, whereas the scale parameter cannot be estimated consistently, thus advising caution in posterior inference. We conclude our work by discussing other SSPs, and presenting some emerging generalizations of SSPs, mostly in biological sciences, which deal with "feature-sampling" problems, multiple populations of individuals sharing species and classes of Markov chains.