Combining classification improvements by ensemble processing

Abstract

The k-nearest neighbor (KNN) classification is a simple and effective classification approach. However, improving performance of the classifier is still attractive. Combining multiple classifiers is an effective technique for improving accuracy. There are many general combining algorithms, such as Bagging, Boosting, or Error Correcting Output Coding that significantly improve the classifier such as decision trees, rule learners, or neural networks. Unfortunately, these combining methods developed do not improve the nearest neighbor classifiers. In this paper, jirst, we present a new approach to combine multiple KNN classi-Jiers based on different distance functions, in which we apply multiple distance functions to improve the performance of the k-nearest neighbor classifiel: Second, we develop a combining method, in which the weights of the distance function, are learnt by genetic algorithm. Finally, combining classifiers in error correcting output coding, are discussed. The proposed algorithms seek to increase generalization accuracy when compared to the basic k-nearest neighbor algorithm. Experiments have been conducted on some benchmark datasets from the UCI Machine Learning Repository. The results show that the proposed algorithms improve the performance of the k-nearest neighbor classification.

Selection–fusion approach for classification of datasets with missing values

“…According to the discussion about the desirable subsets, the number of the features in each subset should be large enough to get a reliable determination of the class label. A lower bound can be obtained using a simple feature reduction technique; for details see [9-10]. …”

“…Generally speaking, there are five classes of well-established strategies to deal with the missing values: 1) discard the incomplete samples (e.g., pairwise deletion [2]); 2) avoid the missing features by dynamic decisions (e.g., decision trees such as CART [7]); 3) recover unknown values from the similar samples (e.g., Expectation Maximization (EM) [8]); 4) insert possible values for the missing features, classify after each insertion and combine the classification results (e.g., Multiple Imputations (MI) [9]); and 5) design multiple classifiers on the subsets of the data and combine the classification results (e.g., ensemble classifiers [17]).…”

“…Multiple imputations (MI) method [1,9] is an alternative solution that uses Monte Carlo simulation to generate more than one imputation of the missing values. However, the MI usually implies several assumptions on the data distribution such as joint normality [13] and regression relationships [14].…”

Loss-Weighted Decoding for Error-Correcting Output Codin

“…In this way, a robust decoding strategy is required to obtain accurate results. Several techniques for the binary decoding step have been proposed in the literature (Windeatt and Ghaderi, 2003) (Ishii et al, 2005) (Passerini et al, 2004) (Dekel and Singer, 2002), though the most common ones are the Hamming and the Euclidean approaches (Windeatt and Ghaderi, 2003). In the work of (Pujol et al, 2006), authors showed that usually the Euclidean distance was more suitable than the traditional Hamming distance in both the binary and the ternary cases.…”

“…In (Windeatt and Ghaderi, 2003), Inverse Hamming Distance (IHD) and Centroid distance (CEN) for binary problems are introduced. Other decoding strategies for nominal, discrete and heterogeneous attributes have been proposed in (Ishii et al, 2005). With the introduction of the zero symbol, Allwein et al (Allwein et al, 2002) show the advantage of using a loss based function of the margin of the base classifier on the ternary ECOC.…”