Pattern Generation through Feature Values Modification and Decision Tree Ensemble Construction

Abstract

Abstract-An ensemble method produces diverse classifiers and combines their decisions for ensemble's decision. A number of methods have been investigated for constructing ensemble in which some of them train classifiers with the generated patterns. This study investigates a new technique of training pattern generation that is easy and effective for ensemble construction. The method modifies feature values of some patterns with the values of other patterns to generate different patterns for different classifiers. The ensemble of decision trees based on the proposed technique was evaluated using a suite of 30 benchmark classification problems, and was found to achieve performance better than or competitive with related conventional methods. Furthermore, two different hybrid ensemble methods have been investigated incorporating the proposed technique of pattern generation with two popular ensemble methods bagging and random subspace method (RSM). It is found that the performance of bagging and RSM algorithms can be improved by incorporating feature values modification with their training processes. Experimental investigation of different types of modification techniques finds that feature values modification with pattern values in the same class is better for generalization.Index Terms-Decision tree ensemble, diversity, feature values modification, generalization, pattern generation. I. INTRODUCTIONThe goal of ensemble construction with several classifiers is to achieve better generalization ability over individual classifiers. The inspiration for building an ensemble is the same as for establishing a committee of people: each member of the committee should be as competent as possible, but the members should be complementary to one another. If the members are not complementary, i.e., if they always agree, the committee is unnecessary as any one member could perform the task of the committee. If the members are complementary, then when one or a few members make an error, there is a high probability that the remaining members can correct his error. Thus, for ensemble construction, proper diversity among classifiers (also called base classifiers) is considered to be an important parameter so that the failure of one may be compensated by others [1], [2].An ensemble method produces diverse classifiers and combines their decisions for ensemble's decision. As a base M. M. Hafizur Rahman is with Dept. of Computer Science, KICT, International Islamic University Malaysia, Jalan Gombak, 50728 Selayang, Selangor, Malaysia (e-mail: hafizur@iium.edu.my).K. Murase is with Graduate School of Engineering, University of Fukui, 3-9-1 Bunkyo, Fukui 910-8507, Japan (e-mail: murase@u-fukui.ac.jp).classifier, decision trees (DTs) are one of the most commonly used methods because they are efficient [3], [4]. Considerable work has been done to determine the effective ways for constructing diverse DTs so that the benefit of ensemble construction could be achieved. There are many ways, such as using different training sets and learning metho...