Can high-order dependencies improve mutual information based feature selection?

Abstract

Mutual information (MI) based approaches are a popular paradigm for feature selection. Most previous methods have made use of low-dimensional MI quantities that are only effective at detecting low-order dependencies between variables. Several works have considered the use of higher dimensional mutual information, but the theoretical underpinning of these approaches is not yet comprehensive. To fill this gap, in this paper, we systematically investigate the issues of employing high-order dependencies for mutual information based feature selection. We first identify a set of assumptions under which the original high-dimensional mutual information based criterion can be decomposed into a set of low-dimensional MI quantities. By relaxing these assumptions, we arrive at a principled approach for constructing higher dimensional MI based feature selection methods that takes into account higher order feature interactions. Our extensive experimental evaluation on real data sets provides concrete evidence that methodological inclusion of highorder dependencies improve MI based feature selection.

Discretization and Feature Selection Based on Bias Corrected Mutual Information Considering High-Order Dependencies

“…For example, JMI [19] criteria can be derived setting the value of β = γ = 1 |S| . In [17], the authors propose a new criterion by relaxing the the first assumption. They show under the relaxed assumption that the selected features are conditionally independent given the f m and another feature f i in S, the redundancy term can be approximated as the following…”

“…Otherwise, f m is discarded considering that it does not contribute to the score significantly. While selecting a new feature, its discretization level is also shifted by a small value δ from its original value (as selected previously based on J rel as shown in line [16][17][18][19][20][21]. This process helps to select the discretization level of features dynamically considering its dependency with other feature.…”

“…However, the computation of I(S; C) is a NP-hard problem [7]. To overcome this problem, different approximations such as MIFS [1], mRMR [10], JMI [19], RelaxMRMR [17] have been proposed over the last decades. In these methods, MI terms such as feature relevancy(R), redundancy(r), conditional redundancy(c) and interaction(i) are considered in order to achieve a better approximation.…”

“…In a recent method mDSM [16], it is shown that incorporating bias correction for R, r, and c terms improves the classification performance. However, the interaction term is not considered in mDSM which needs to be addressed for better approximation [17].…”

Use of relevancy and complementary information for discriminatory gene selection from high-dimensional cancer data

“…107 Since MI based filter methods do not extract new features and thus are more 108 interpretable, parallel to the development in Deep learning, there has been a lot of effort 109 to better approximate MI measures such as relevancy and redundancy. New Information 110 theoretic measures such as complementary information, the additional information that 111 a gene has about the class, which is not found in the already selected subset of genes 112 have been proposed [15,27]. These methods attempt to estimate the joint mutual 113 information of a feature subset with the class.…”