Robust Bayesian Linear Classifier Ensembles

Abstract

Abstract. Ensemble classifiers combine the classification results of several classifiers.Simple ensemble methods such as uniform averaging over a set of models usually provide an improvement over selecting the single best model. Usually probabilistic classifiers restrict the set of possible models that can be learnt in order to lower computational complexity costs. In these restricted spaces, where incorrect modelling assumptions are possibly made, uniform averaging sometimes performs even better than bayesian model averaging. Linear mixtures over sets of models provide an space that includes uniform averaging as a particular case. We develop two algorithms for learning maximum a posteriori weights for linear mixtures, based on expectation maximization and on constrained optimizition. We provide a nontrivial example of the utility of these two algorithms by applying them for one dependence estimators.We develop the conjugate distribution for one dependence estimators and empirically show that uniform averaging is clearly superior to BMA for this family of models. After that we empirically show that the maximum a posteriori linear mixture weights improve accuracy significantly over uniform aggregation.

To Select or To Weigh: A Comparative Study of Model Selection and Model Weighing for SPODE Ensembles

“…It has also been shown that a SPODE ensemble can further improve upon the classification accuracy of a single SPODE by decreasing the classification variance [13,18]. The first approach to ensembling SPODEs was AODE [13] which used equal weight combination of all SPODEs whose parent occurred with a user-specified minimum frequency in the training data.…”

“…The first approach to ensembling SPODEs was AODE [13] which used equal weight combination of all SPODEs whose parent occurred with a user-specified minimum frequency in the training data. Subsequent research suggested that frequency is not a useful model selection criterion and that appropriate weighting can substantially improve upon equal weighting, proposing weighting schemes such as MAPLMG [18]. On the other hand, it has also been shown that model selection can be very effective when ensembling SPODEs [22].…”

“…Among alternative semi-naive forms, Super-Parent-One-Dependence Estimators (SPODEs) [2,3], and particularly ensembles thereof [13] have received a lot of attention [18,19,20,21,22] because they offer a combination of high training efficiency, high classification efficiency and high classification accuracy. Those merits give SPODEs a great potential to substitute for naive Bayes classifiers in numerous real-world classification systems, including medical diagnosis, fraud detection, email filtering, document classification and webpage prefetching.…”

Finding the Right Family: Parent and Child Selection for Averaged One-Dependence Estimators

“…, x n the original AODE excludes ODEs with parent x i where the frequency of the value x i is lower than limit m=30, a widely used minimum on sample size for statistical inference purposes. However, subsequent research [14] shows that this constraint actually increases error and hence the current research uses m=1.…”

“…Semi-naive Bayesian techniques further improve naive Bayes' accuracy by relaxing its assumption that the attributes are conditionally independent [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]. One approach to weakening this assumption is to use an x-dependence classifier [7], in which each attribute depends upon the class and at most x other attributes.…”