Multi-Objective Evolutionary Algorithms for Filter Based Feature Selection in Classification

Abstract

Feature selection is a multi-objective problem with the two main conflicting objectives of minimising the number of features and maximising the classification performance. However, most existing feature selection algorithms are single objective and do not appropriately reflect the actual need. There are a small number of multi-objective feature selection algorithms, which are wrapper based and accordingly are computationally expensive and less general than filter algorithms. Evolutionary computation techniques are particularly suitable for multi-objective optimisation because they use a population of candidate solutions and are able to find multiple non-dominated solutions in a single run. However, the two well-known evolutionary multi-objective algorithms, nondominated sorting based multi-objective genetic algorithm II (NSGAII) and strength Pareto evolutionary algorithm 2 (SPEA2) have not been applied to filter based feature selection. In this work, based on NSGAII and SPEA2, we develop two multi-objective, filter based feature selection frameworks. Four multi-objective feature selection methods are then developed by applying mutual information and entropy as two different filter evaluation criteria in each of the two proposed frameworks. The proposed multi-objective algorithms are examined and compared with a single objective method and three traditional methods (two filters and one wrapper) on eight benchmark datasets. A decision tree is employed to test the classification performance. Experimental results show that the proposed multi-objective algorithms can automatically evolve a set of non-dominated solutions that include a smaller number of features and achieve better classification performance than using all features. NSGAII and SPEA2 outperform the single objective algorithm, the two traditional filter algorithms and even the traditional wrapper algorithm in terms of both the number of features and the classification performance in most cases. NSGAII achieves similar performance to SPEA2 for the datasets that consist of a small number of features and slightly better results when the number of features is large. This work represents the first study on NSGAII and SPEA2 for filter feature selection in classification problems with both providing field leading classification performance.

A Survey on Evolutionary Computation Approaches to Feature Selection

“…In addition, Section IV discusses the research on EC based filter approaches for feature selection. The applications of EC for feature selection are described in Section V. [37], [58], [38], [39], [44], [59], [60], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79], [80], [81], [82], [83], [84], [85], [86], [87] [88], [89], [90], [91], [92], [93], [94], [95], [96], [97] Filter [75], [98], [99], [100], [101], [102] [102],…”

“…For filter approaches, different measures have been applied to GAs for feature selection, e.g. information theory [102], [105], [106], consistency measures [98], [105], rough set theory [103] and fuzzy set theory [99].…”

“…GAs for multi-objective feature selection started much later (around 10 years later) than for single objective feature selection. Most of the multi-objective approaches are based on non-dominated sorting GA II (NSGA-II) or its variations [92], [94], [96], [97], [102], [103], [105]. Although there are more works on multi-objective feature selection using GAs than using other EC techniques, the potential of GAs for multi-objective feature selection has still not been thoroughly investigated since feature selection is a complex task that requires specifically designed multi-objective GAs to search for the non-dominated solutions.…”

“…This is considered the most popular way to use information theory for feature selection. Based on the idea of "Max-relevance and min-redundancy" [32], [33], [34], [41], [50], [55], [72], [102], [105], [106], [116], [118] [173], [177], [195], [201], [202], [203], [205], [229] Correlation Measure [75], [107], [116], [203], [208] Distance Measure [11], [15], [118], [136], [171] Consistency Measure [98], [105], [166], [203] Fuzzy Set Theory [99], [104], [164], [166], [169] Rough Set Theory [103], [163], [167], [168], [170], [172], [174], [183], [197],…”

Filter based backward elimination in wrapper based PSO for feature selection in classification

“…Experiments show that PSO using mutual information can effective address feature selection problems. Our recent work on PSO for feature selection can be seen in [25,26,27,28] Based on PSO and a statistical clustering method [29,30] that groups features to different clusters and similar features to the same cluster, Lane et al [31] proposed a feature selection algorithm, which uses PSO to select one feature from each cluster. The results show that by selecting a representative feature from each cluster, the proposed algorithm can significantly reduce the number of features and increase the classification performance.…”