ReliefE: feature ranking in high-dimensional spaces via manifold embeddings

Abstract

Feature ranking has been widely adopted in machine learning applications such as high-throughput biology and social sciences. The approaches of the popular Relief family of algorithms assign importances to features by iteratively accounting for nearest relevant and irrelevant instances. Despite their high utility, these algorithms can be computationally expensive and not-well suited for high-dimensional sparse input spaces. In contrast, recent embedding-based methods learn compact, low-dimensional representations, potentially facilitating down-stream learning capabilities of conventional learners. This paper explores how the Relief branch of algorithms can be adapted to benefit from (Riemannian) manifold-based embeddings of instance and target spaces, where a given embedding’s dimensionality is intrinsic to the dimensionality of the considered data set. The developed ReliefE algorithm is faster and can result in better feature rankings, as shown by our evaluation on 20 real-life data sets for multi-class and multi-label classification tasks. The utility of ReliefE for high-dimensional data sets is ensured by its implementation that utilizes sparse matrix algebraic operations. Finally, the relation of ReliefE to other ranking algorithms is studied via the Fuzzy Jaccard Index.

Multi-institutional prognostic modeling of survival outcomes in NSCLC patients treated with first-line immunotherapy using radiomics

“… RL, MSF, and SF: The Relief algorithm is a popular and effective feature selection method in machine learning. It was originally proposed by Kira and Rendell in 1992 [ 27 , 28 ]. It is among the most widely used non-myopic algorithms for feature ranking, where each feature is assigned a real-valued score, offering insights into its importance [ 28 ].…”

“…It was originally proposed by Kira and Rendell in 1992 [ 27 , 28 ]. It is among the most widely used non-myopic algorithms for feature ranking, where each feature is assigned a real-valued score, offering insights into its importance [ 28 ]. Extensions and variations of the Relief algorithm have been proposed to address specific challenges or enhance its performance.…”

Pruning Decision Rules by Reduct-Based Weighting and Ranking of Features

“…The minimum value of the factor is zero when none of reducts in the set contains the given variable. Therefore, unlike some other ranking mechanisms, such as Relief [ 31 ], with the proposed weighting factor, some attributes can be considered irrelevant and assigned the rank of zero. The maximum value would be found for all reducts in the set being of the same length l , and all including the attribute.…”

“…All variables are considered as relevant but of course with observed degrees of relevance. Relief and its variants are popular examples of this category of methods [ 31 ]. The second group of approaches can consider some features as completely irrelevant and assign to them the rank of zero.…”

Multi-view Self-attention for Regression Domain Adaptation with Feature Selection

“…In this section we explore the use of attention-based neural networks mechanism for estimating feature importance in domain adaptation. This section is inspired from the seminal works on the attention mechanism [3,34,29]. We took inspiration from [29] regarding feature importance, with a different implementation of the attention mechanism, we defined it as follows:…”