Minimising Added Classification Error Using Walsh Coefficients

Abstract

Abstract. Two-class supervised learning in the context of a classifier ensemble may be formulated as learning an incompletely specified Boolean function, and the associated Walsh coefficients can be estimated without knowledge of the unspecified patterns. Using an extended version of the Tumer-Ghosh model, the relationship between Added Classification Error and second order Walsh coefficients is established. In this paper, the ensemble is composed of Multi-layer Perceptron (MLP) base classifiers, with the number of hidden nodes and epochs systematically varied. Experiments demonstrate that the mean second order coefficients peak at the same number of training epochs as ensemble test error reaches a minimum.

Approximation of Ensemble Boundary Using Spectral Coefficients

“…1 b is the amount that the k th classifier boundary ( ) differs from the ideal Bayes boundary. Assuming that b is a Gaussian random variable, closed-form expressions may be obtained for [17] [20]. Further details about the model and assumptions may be found in [20].…”

“…This mapping may be analysed using Walsh spectral coefficients, which was first proposed for pattern recognition over four decades ago [16], although not in the context of ensemble classification. The relationship between added classification error and second order Walsh coefficients was established in [17]. In [13] first and second order Walsh coefficients were used for ensemble pruning.…”

Low Training Strength High Capacity Classifiers for Accurate Ensembles Using Walsh Coefficients

“…Mutually exclusive areas under the probability distribution are labelled 1 -8 in Figure 2, and denoting the number of patterns in area y by a y , the contribution from classifiers i,j according to area is given in Table 1. The model assumptions are discussed in [3], in which the expression for the difference in Added Classification Error of ith and jth classifiers Averaging over all pairs of classifiers in (7) the mean difference in added error is given by…”

“…In [3] it is shown that S2M is a good predictor of ensemble performance as number of epochs is increased. For the datasets in Section 4, optimal performance for majority vote occurs on average around 2-3 epochs.…”

“…First order Walsh coefficients were shown to provide a measure of class separability for selecting optimal base classifiers in [2], in which it is also shown that this does not imply optimality of the ensemble. In contrast, in [3] it was shown that second order Walsh coefficients may be used to determine optimal ensemble performance. The motivation for using Walsh coefficients in ensemble design is fully explored in [4] and [2].…”