AFS: An Attention-Based Mechanism for Supervised Feature Selection

Abstract

As an effective data preprocessing step, feature selection has shown its effectiveness to prepare high-dimensional data for many machine learning tasks. The proliferation of high dimension and huge volume big data, however, has brought major challenges, e.g. computation complexity and stability on noisy data, upon existing feature-selection techniques. This paper introduces a novel neural network-based feature selection architecture, dubbed Attention-based Feature Selection (AFS). AFS consists of two detachable modules: an attention module for feature weight generation and a learning module for the problem modeling. The attention module formulates correlation problem among features and supervision target into a binary classification problem, supported by a shallow attention net for each feature. Feature weights are generated based on the distribution of respective feature selection patterns adjusted by backpropagation during the training process. The detachable structure allows existing off-theshelf models to be directly reused, which allows for much less training time, demands for the training data and requirements for expertise. A hybrid initialization method is also introduced to boost the selection accuracy for datasets without enough samples for feature weight generation. Experimental results show that AFS achieves the best accuracy and stability in comparison to several state-of-art feature selection algorithms upon both MNIST, noisy MNIST and several datasets with small samples.

Designing a Sensor Glove Using Deep Learning

“…Afterwards, they analysed the selected features for any relation, however this required prior knowledge on the subject, which is not always readily available. A variation of this concept by Ning Gui et al [7], employs a detachable NN between the input and the model, where they attached an attention network to a trained NN. The detachable NN, the attention network, selects features from the input using the correlation within to generate probability for each feature.…”

“…The basis of the method is from Yifeng et al [16] and Ning Gui, et al [7], introducing a sparse layer between the input and the first hidden layer of a neural network, to then perform additional processing for the model to select a subset of features during training. Our approach expands upon such in two areas: feature analysis and feature selection.…”

“…Our feature selection method builds upon that from Yifeng et al [16] and Ning Gui et al [7]. But it is significantly different from both.…”

“…Ning Gui et al [7] used an attention network between the input and the model, utilising the correlation between the features to determine which features are important based on their probability. Our method does not require the deployment of an attention network, but a layer of weights, which is easier to implement and less complex to understand.…”

Which Attributes Matter the Most for Loan Origination? A Neural Attention Approach

“…Then, we utilise this information, so that we can better exploit the existent information, to better inform the decision making of the model. So in this paper, we are trying to show another side of these models, the one that focuses on the attention mechanism, and thus extracting the feature importance of the data Ning Gui (2019).…”