Methods of combining multiple classifiers and their applications to handwriting recognition

Abstract

Method of combining the classification powers of several classifiers is regarded as a general problem in various application areas of pattern recognition, and a systematic investigation has been made. Possible solutions to the problem can be divided into three categories according to the levels of information available from the various classifiers. Four approaches are proposed based on different methodologies for solving this problem. One is suitable for combining individual classifiers such as Bayesian, k-NN and various distance classifiers. The other three could be used for combining any kind of individual classifiers. On applying these methods to combine several classifiers for recognizing totally unconstrained handwritten numerals, the experimental results show that the performance of individual classifiers could be improved significantly. For example, on the U.S. zipcode database, the result of 98.9% recognition with 0.90% substitution and 0.2% rejection can be obtained, as well as a high reliability with 95% recognition, 0% substitution and 5% rejection. These results compared favorably to other research p u p s in Europe, Asia, and North America.

On evaluating brain tissue classifiers without a ground truth

“…Let θ j be an l × l matrix. Each element θ j (t′, t) describes the probability that rater j labels a voxel with t ′ when the true label is t. This matrix is similar to the normalized confusion matrix of a Bayesian classifier (Xu et al, 1992), and we will use this terminology for the remainder of the paper. Let θ= [θ 1 , …, θ r ] be the unknown set of all confusion matrices characterizing all r raters.…”

Ensembles of Learning Machines

“…This approach can be refined assigning different weights to each classifier to optimize the performance of the combined classifier on the training set [86], or, assuming mutual independence between classifiers, a Bayesian decision rule selects the class with the highest posterior probability computed through the estimated class conditional probabilities and the Bayes' formula [130,122]. A Bayesian approach has also been used in Consensus based classification of multisource remote sensing data [10,9,19], outperforming conventional multivariate methods for classification.…”

“…Feature subspace methods performed by partitioning the set of features, where each subset is used by one classifier in the team, are proposed in [130,99,18]. Other methods for combining different feature sets using genetic algorithms are proposed in [81,79].…”

“…In this last decade one of the main research areas in machine learning has been represented by methods for constructing ensembles of learning machines. Although in the literature [86,129,130,69,61,23,33,12,7,37] a plethora of terms, such as committee, classifier fusion, combination, aggregation and others are used to indicate sets of learning machines that work together to solve a machine learning problem, in this paper we shall use the term ensemble in its widest meaning, in order to include the whole range of combining methods. This variety of terms and specifications reflects the absence of an unified theory on ensemble methods and the youngness of this research area.…”

Combining predictions in pairwise classification: An optimal adaptive voting strategy and its relation to weighted voting

“…A large number of strategies have been proposed for this purpose, see for example [5][6][7][8][9]. Since the aggregation problem also occurs in all other decomposition methods and in ensemble methods, these research areas as well provide a large number of aggregation strategies (sometimes called classifier combination schemes); see for example [10] and references therein. However, since the semantics of these problems are different, we note that the aggregation strategies from different fields are not always interchangeable.…”