Feature Selection in Regression Tasks Using Conditional Mutual Information

Abstract

Abstract. This paper presents a supervised feature selection method applied to regression problems. The selection method uses a Dissimilarity matrix originally developed for classification problems, whose applicability is extended here to regression and built using the conditional mutual information between features with respect to a continuous relevant variable that represents the regression function. Applying an agglomerative hierarchical clustering technique, the algorithm selects a subset of the original set of features. The proposed technique is compared with other three methods. Experiments on four data-sets of different nature are presented to show the importance of the features selected from the point of view of the regression estimation error (using Support Vector Regression) considering the Root Mean Squared Error (RM SE).

Filter-Type Variable Selection Based on Information Measures for Regression Tasks

“…The contributions of this paper are two-fold: (a) to establish a methodology to properly solve the estimation of this distance for regression problems where the relevant variable is continuous, through the assessment of the conditional mutual information between input and output variables; and (b) to show the extension of this methodology to multi-output regression datasets. Some preliminary results were presented in [12], where the method was applied only to single-output regression datasets. In addition, the work presented here introduces an information theoretic framework for the distance used in the clustering-based feature selection process in regression tasks, when using continuous variables.…”

A real estate valuation model using boosted feature selection

“…By contrast, it was shown that the mutual information criterion fails to select optimal features in some situations. Feature clustering-based methods aiming to find a subset of the features that minimizes the regression error using conditional MI has been proposed ( [45], [46], [47]) as an approach for the application of MI in regression problems. These studies show the efficacy of MI to perform in regression task with suitable stopping criteria.…”