Quick and robust feature selection: the strength of energy-efficient sparse training for autoencoders

Abstract

Major complications arise from the recent increase in the amount of high-dimensional data, including high computational costs and memory requirements. Feature selection, which identifies the most relevant and informative attributes of a dataset, has been introduced as a solution to this problem. Most of the existing feature selection methods are computationally inefficient; inefficient algorithms lead to high energy consumption, which is not desirable for devices with limited computational and energy resources. In this paper, a novel and flexible method for unsupervised feature selection is proposed. This method, named QuickSelection (The code is available at: https://github.com/zahraatashgahi/QuickSelection), introduces the strength of the neuron in sparse neural networks as a criterion to measure the feature importance. This criterion, blended with sparsely connected denoising autoencoders trained with the sparse evolutionary training procedure, derives the importance of all input features simultaneously. We implement QuickSelection in a purely sparse manner as opposed to the typical approach of using a binary mask over connections to simulate sparsity. It results in a considerable speed increase and memory reduction. When tested on several benchmark datasets, including five low-dimensional and three high-dimensional datasets, the proposed method is able to achieve the best trade-off of classification and clustering accuracy, running time, and maximum memory usage, among widely used approaches for feature selection. Besides, our proposed method requires the least amount of energy among the state-of-the-art autoencoder-based feature selection methods.

Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning

“…Feature selection aims to find the most relevant features from the input and is studied extensively in the machine-learning literature. Approaches can be grouped according to whether labeled data is used-supervised (Nie et al, 2010) or unsupervised (Ball & Hall, 1965;Hilborn & Lainiotis, 1967;He et al, 2005;Balın et al, 2019;Atashgahi et al, 2020)-or what high-level approach is takenfilter methods (Blum & Langley, 1997), wrapper methods (Kohavi & John, 1997), or embedded methods (Yuan & Lin, 2006). Most relevant to our work are embedded supervised methods as they have good scaling properties while achieving the best performance which is vital in our setting with over a million features.…”

Efficient Intrusion Detection System in the Cloud Using Fusion Feature Selection Approaches and an Ensemble Classifier

“…In an AE, the data are reconstructed after computations based on the concept of learning the best features and matching the output to the input as closely as feasible [45]. Stacked AEs are one of the AE types designed for automated feature selection [46]. In an SAE, the output of the first AE is the input of the second one.…”

A brain-inspired algorithm for training highly sparse neural networks

“…In Mostafa and Wang (2019), the authors proposed the idea of parameter reallocation automatically across layers during sparse training in CNNS. Many works have further studied sparse training concept recently (Atashgahi et al, 2022;Gordon et al, 2018;Liu et al, 2020Liu et al, , 2021a.…”

“…Sparse neural networks have been considered as an effective solution to address these challenges (Hoefler et al, 2021;Mocanu et al, 2021). By using sparsely connected layers instead of fully-connected ones, sparse neural networks have reached a competitive performance to their dense equivalent networks in various applications (Frankle & Carbin, 2018;Atashgahi et al, 2022), while having much fewer parameters. It has been shown that biological brains, especially the human brain, enjoy sparse connections among neurons (Friston, 2008).…”