Ensemble feature selection in medical datasets: Combining filter, wrapper, and embedded feature selection results

Abstract

Feature selection is a process aimed at filtering out unrepresentative features from a given dataset, usually allowing the later data mining and analysis steps to produce better results. However, different feature selection algorithms use different criteria to select representative features, making it difficult to find the best algorithm for different domain datasets. The limitations of single feature selection methods can be overcome by the application of ensemble methods, combining multiple feature selection results. In the literature, feature selection algorithms are classified as filter, wrapper, or embedded techniques. However, to the best of our knowledge, there has been no study focusing on combining these three types of techniques to produce ensemble feature selection. Therefore, the aim here is to answer the question as to which combination of different types of feature selection algorithms offers the best performance for different types of medical data including categorical, numerical, and mixed data types. The experimental results show that a combination of filter (i.e., principal component analysis) and wrapper (i.e., genetic algorithms) techniques by the union method is a better choice, providing relatively high classification accuracy and a reasonably good feature reduction rate.

Ensemble Feature Ranking for Cost-Based Non-Overlapping Groups: A Case Study of Chronic Kidney Disease Diagnosis in Developing Countries

“…Although both multi-point intersection and union result in solutions having comparable accuracy, the former technique tends to select a feature set with lower cardinality as compared with the latter technique. This empirical observation is supported by several works where the majority vote option is preferred over others [10], [27], [29].…”

“…One of the key limitations of these methods is ignoring feature interaction. Methods that take into [20], [29], [30]. These ensemble techniques have taken inspiration from ensemble model learning where weak learners provide intermediate labels that are subsequently combined into a final recommendation.…”

“…In this regard, the final feature subset is based on the consolidation of the base subsets. Typical ways to arrive at a final feature subset is to use set-theoretic combining techniques such as the intersection of all the base feature subsets, or union of all the subsets or multi-point intersection is also one of the favored techniques in this regard [29]. It is reported that intersection tends to produce very restrictive solutions that do not tend to produce good results [31].…”

“…All the experiments are performed on the UCI benchmarked CKD dataset. Tsai and Hsiao [29] performed an extensive study on feature subset combination methods such as union, intersection, and multi-intersection approaches. The main objective of this paper is to explore the effective approaches to combine candidate solutions from multiple feature selection methods.…”

“…Statistical Difference Chen et al, [29] * Ali et al, [30] * Salekin et al, [22] * Polat et al, [13] * Almansour et al, [21] * Ogunley et al, [10] † Qin et al, [12] * Osanaiye et al, [28] * * Proposed solution is not statistically better † Proposed solution is statistically better solution-2 but the overall cost is also low. It is important to note that all the results reported in Table 18 are based on the CKD diagnosis dataset.…”