Feature Selection with Conditional Mutual Information Considering Feature Interaction

Abstract

Feature interaction is a newly proposed feature relevance relationship, but the unintentional removal of interactive features can result in poor classification performance for this relationship. However, traditional feature selection algorithms mainly focus on detecting relevant and redundant features while interactive features are usually ignored. To deal with this problem, feature relevance, feature redundancy and feature interaction are redefined based on information theory. Then a new feature selection algorithm named CMIFSI (Conditional Mutual Information based Feature Selection considering Interaction) is proposed in this paper, which makes use of conditional mutual information to estimate feature redundancy and interaction, respectively. To verify the effectiveness of our algorithm, empirical experiments are conducted to compare it with other several representative feature selection algorithms. The results on both synthetic and benchmark datasets indicate that our algorithm achieves better results than other methods in most cases. Further, it highlights the necessity of dealing with feature interaction.

Genetic Algorithm for the Mutual Information-Based Feature Selection in Univariate Time Series Data

“…This section shows and analyzes the experimental results of our design of the GA for solving the feature selection problem. We compare the performance of the proposed work with two existing heuristics: (i) Basic Open-source Nonlinear Mixed INteger programming (BONMIN) [26], [27], and, (ii) CMIFSI [11]. BONMIN is a popular open-source tool for solving Mixed Integer Non-Linear Programming problems, and is a part of several widely used optimization integrated development environments (IDEs) such as AMPL IDE [27].…”

“…BONMIN is a popular open-source tool for solving Mixed Integer Non-Linear Programming problems, and is a part of several widely used optimization integrated development environments (IDEs) such as AMPL IDE [27]. The CMFSI is a feature selection method based on conditional mutual information proposed recently by Liang et al [11]. Two time-series data sets from a public repository, Machine Learning Mastery [28] were used.…”

“…In the classifier, the number of attributes (i.e., the maximum features) is not huge. The outcome of the forecaster has a vast range, whereas, in classifiers, the number of classes is a small number, such as from 3 to 10 [11].…”

“…In CMI-based methods, the objective function contains terms to account for the relevancy and redundancy of features. Liang et al pointed out that that interaction among features is also common in classification data sets [11]. Therefore, the objective function should also include a term to account for the interaction of any feature with the selection set.…”

“…The number of features is usually large, and therefore, the problem becomes intractable. Both constructive and iterative heuristics are useful in solving the feature selection problem [3], [9]- [11]. In practice, data often contains noise or missing values.…”