Learning by extrapolation from marginal to full-multivariate probability distributions: decreasingly naive Bayesian classification

Abstract

Averaged n-Dependence Estimators (AnDE) is an approach to probabilistic classification learning that learns by extrapolation from marginal to full-multivariate probability distributions. It utilizes a single parameter that transforms the approach between a lowvariance high-bias learner (Naive Bayes) and a high-variance low-bias learner with Bayes optimal asymptotic error. It extends the underlying strategy of Averaged One-Dependence Estimators (AODE), which relaxes the Naive Bayes independence assumption while retaining many of Naive Bayes' desirable computational and theoretical properties. AnDE further relaxes the independence assumption by generalizing AODE to higher-levels of dependence. Extensive experimental evaluation shows that the bias-variance trade-off for Averaged 2-Dependence Estimators results in strong predictive accuracy over a wide range of data sets. It has training time linear with respect to the number of examples, learns in a single pass through the training data, supports incremental learning, handles directly missing values, and is robust in the face of noise. Beyond the practical utility of its lower-dimensional variants, AnDE is of interest in that it demonstrates that it is possible to create low-bias high-variance generative learners and suggests strategies for developing even more powerful classifiers.

$$\text {ALR}^n$$ ALR n : accelerated higher-order logistic regression

“…This hypothesis was tested in the context of Bayesian Network classifiers in Webb et al (2011Webb et al ( , 2005 where the results corroborated the hypothesis. However, we are not aware of any past work that investigates this hypothesis in the context of higher-order Logistic Regression.…”

“…It has been shown that Bayesian Network Classifiers (BNCs) that explicitly represent higher-order interactions tend to have lower bias than those that do not (Martinez et al 2016;Webb et al 2011). This is because BNCs that can represent higher-order interactions can exactly represent any of a superset of the distributions that can be represented by BNCs that are restricted to lower order interactions.…”

Non-Disjoint Discretization for Aggregating One-Dependence Estimator Classifiers

“…Still, we plan to extend the experimental part to a test bed of high dimensional datasets in order to corroborate these conclusions. Moreover, we believe that the positive results observed in AODE are a good motivation to think that the beneficial properties of NDD will be strengthen when applied to Aggregating n-dependence estimators (AnDE) [19], for values of n greater or equal to 2 (since when n = 1 it is equivalent to AODE).…”

K-Dependence Bayesian Classifier Ensemble

“…Unfortunately there is no a priori means to preselect an appropriate value of k that can help to achieve the lowest error for a given training set, as this is a complex interplay between the data quantity and the complexity and strength of the interactions between the attributes proved by Martinez et al [8]. From the discussion above, we can see that, for each KDF i , the space complexity of the probability table increases exponentially as k increases; to achieve the trade-off between classification performance and efficiency, we restrict the structure complexity to be two-dependence, which is also adopted by Webb et al [26].…”