A Survey on Bayesian Nonparametric Learning

Abstract

Bayesian (machine) learning has been playing a significant role in machine learning for a long time due to its particular ability to embrace uncertainty, encode prior knowledge, and endow interpretability. On the back of Bayesian learning's great success, Bayesian nonparametric learning (BNL) has emerged as a force for further advances in this field due to its greater modelling flexibility and representation power. Instead of playing with the fixed-dimensional probabilistic distributions of Bayesian learning, BNL creates a new "game" with infinite-dimensional stochastic processes. BNL has long been recognised as a research subject in statistics and, to date, several state-of-the-art pilot studies have demonstrated that BNL has a great deal of potential to solve real-world machine learning tasks. However, despite these promising results, BNL has not created a huge wave in the machine learning community. Esotericism may account for this. The books and surveys on BNL written by statisticians are overcomplicated and filled with tedious theories and proofs. Each is certainly meaningful but may scare away new researchers, especially those with computer science backgrounds. Hence, the aim of this paper is to provide a plain-spoken, yet comprehensive, theoretical survey of BNL in terms that researchers in the machine learning community can understand. It is hoped this survey will serve as a starting point for understanding and exploiting the benefits of BNL in our current scholarly endeavours. To achieve this goal, we have collated the extant studies in this field and aligned them with the steps of a standard BNL procedure-from selecting the appropriate stochastic processes, through manipulation, to executing the model inference algorithms. At each step, past efforts have been thoroughly summarised and discussed. In addition, we have reviewed the common methods for implementing BNL in various machine learning tasks along with its diverse applications in the real-world as examples to motivate future studies.

Bayesian deep learning methods applied to diabetic retinopathy disease: a review

“…On the other hand, to address various types of model uncertainties, BDL is concerned with determining the posterior distribution of weights. BDL has several benefits over classical DL [43]. Any shape of probabilistic distributions like Gaussian, beta, gamma or exponential can take the likelihood and prior distributions.…”

A Review on Bayesian Deep Learning in Healthcare: Applications and Challenges

“…In later work, these topics are covered: DL architectures, probabilistic graphical models, BDL in detail, as well as some models and applications of BDL. In 2019, Xuan et al [17] published a review in this manner and covered some definitions of the field, stochastic process and its manipulation, posterior inference, application for machine learning tasks, and some real-world applications. In 2020, Charnock et al [18] published a paper on BDL that addressed various types of uncertainty, Bayesian neural networks with applicable methods, practical implementations of BDL in two approaches (numerically using MCMC, and approximation methods).…”

“…On the other hand, BDL, ''which can also be called a Bayesian neural network,'' is concerned with estimating the posterior distribution of data, especially to deal with different kinds of uncertainties. BDL has several advantages compared to classical deep learning [17]. First, in addition to dealing with overfitting, particularly when the data is insufficient to feed the model, BDL is used to represent and quantify uncertainties of DL models based on the probabilistic foundations of Bayesian statistics.…”

Consistent Estimation of Identifiable Nonparametric Mixture Models from Grouped Observations

“…Much of the literature concerning NoMMs falls in the category of Bayesian nonparametrics, a thorough summary of which is given in [8]. Typically, mixture models in this setting do not assume that the number of mixture components is known, and instead assume that the mixture components are from a known parametric family of distributions.…”