Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy

Abstract

Feature selection is an important problem for pattern classification systems. We study how to select good features according to the maximal statistical dependency criterion based on mutual information. Because of the difficulty in directly implementing the maximal dependency condition, we first derive an equivalent form, called minimal-redundancy-maximal-relevance criterion (mRMR), for first-order incremental feature selection. Then, we present a two-stage feature selection algorithm by combining mRMR and other more sophisticated feature selectors (e.g., wrappers). This allows us to select a compact set of superior features at very low cost. We perform extensive experimental comparison of our algorithm and other methods using three different classifiers (naive Bayes, support vector machine, and linear discriminate analysis) and four different data sets (handwritten digits, arrhythmia, NCI cancer cell lines, and lymphoma tissues). The results confirm that mRMR leads to promising improvement on feature selection and classification accuracy.

A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula

“…It can be used for analyzing behavioral data, where many of the properties we have highlighted could be useful (e.g., CMI, interactions). It has been used for feature selection in general classification problems [Lefakis and Fleuret, 2014; Peng et al, 2005; Torkkola, 2003] and we hope GCMI would provide practical advantages in many such applications. We further suggest that the copula normalization could be used as a general preprocessing step that would convert any covariance‐based statistic or algorithm into a robust rank‐based version (e.g., common spatial patterns, canonical correlation analysis, linear/quadratic discriminant analysis).…”

GEO-CEOS stage 4 validation of the Satellite Image Automatic Mapper lightweight computer program for ESA Earth observation level 2 product generation – Part 2: Validation

“…The proposed ensemble of TQ 2 I summary statistics, specifically, CVPAI2(R: A ⇒ B ⊆ A × B), OA(OAMTRX = FrequencyCount(A × B)) and class-conditional probabilities(OAMTRX), is an original minimally dependent and maximally informative (mDMI) set (Si Liu, Hairong Liu, Latecki, Xu, & Lu, 2011; Peng, Long, & Ding, 2005) of outcome Q 2 Is (O-Q 2 Is), to be jointly maximized according to the Pareto formal analysis of multi-objective optimization problems (Boschetti, Flasse, & Brivio, 2004); refer to the Part 1, Chapter 1.…”

“…According to the GEO-CEOS Val guidelines (GEO-CEOS, 2010; GEO-CEOS WGCV, 2015), Val is the process of assessing, by independent means, the quality of an information processing system by means of an mDMI set (Si Liu et al, 2011; Peng et al, 2005) of community-agreed outcome and process (OP) Q 2 Is (OP- Q 2 Is), each one provided with a degree of uncertainty in measurement, ± δ, with δ ≥ 0%.…”

“…An mDMI set of O-Q 2 Is (Si Liu et al, 2011; Peng et al, 2005), comprising OA(OAMTRX = FrequencyCount(A × B)), CVPAI2(R: A ⇒ B ⊆ A × B), class-conditional probabilities p ( r | t ) of reference class r  =  1 , …, RC =  |B|, given test class t  = 1, …, TC  = |A|, and class-conditional probabilities p ( t | r ), with r  =  1 , …, RC =  |B|, t  = 1, …, TC  = |A|, was estimated in the four test cases described in previous Chapter 4.3 to Chapter 4.5.…”

Applying the Fisher score to identify Alzheimer's disease-related genes

“…Gene selection mainly has two merits (Peng et al, 2005;Saeys et al, 2007). First, it can reduce dramatically the number of genes used in classifying the disease and overcome the problem of the "curse of dimensionality".…”