Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoders

Abstract

Major complications arise from the recent increase in the amount of highdimensional data, including high computational costs and memory requirements. Feature selection, which identifies the most relevant and informative attributes of a dataset, has been introduced as a solution to this problem. Most of the existing feature selection methods are computationally inefficient; inefficient algorithms lead to high energy consumption, which is not desirable for devices with limited computational and energy resources. In this paper, a novel and flexible method for unsupervised feature selection is proposed. This method, named QuickSelection 1 , introduces the strength of the neuron in sparse neural networks as a criterion to measure the feature importance. This criterion, blended with sparsely connected denoising autoencoders trained with the sparse evolutionary training procedure, derives the importance of all input features simultaneously. We implement Quick-Selection in a purely sparse manner as opposed to the typical approach of using a binary mask over connections to simulate sparsity. It results in a considerable speed increase and memory reduction. When tested on several benchmark datasets, including five low-dimensional and three high-dimensional datasets, the proposed method is able to achieve the best trade-off of classification and clustering accuracy, running time, and maximum memory usage, among widely used approaches for feature selection. Besides, our proposed method requires the least amount of energy among the state-of-the-art autoencoder-based feature selection methods.

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration

“…However, pruning plasticity drops significantly after the second learning rate decay, leading to a situation where the pruned networks can not recover from the continued training. This finding helps to explain several observations (1) for gradual magnitude pruning (GMP), it is always optimal to end pruning before the second learning rate drop [73,13]; (2) dynamic sparse training (DST) benefits from a monotonically decreasing pruning rate with cosine or linear update schedule [8,9];…”

“…Our paper re-emphasizes the great potential of during-training pruning in reducing the training/inference resources required by ML models without sacrificing accuracy. It has a significant environmental impact on reducing the energy cost of the ML models and CO2 emissions [1,52,15,55,60]. Pruning rate=0.2 Pruning rate=0.5 Pruning rate=0.9 Pruning rate=0.98…”

“…The implementation used in the paper is modified based on the open-source code of Sparse Momentum repository 1 introduced by [8]. We added VGG-19 with batchnorm from the GraSP repository 2 .…”

Dynamic Sparse Training for Deep Reinforcement Learning

“…Dynamic sparse training has emerged and showed its success in many other fields as well [21,36]. Atashgahi et al [3] adopted the SET algorithm for feature selection and showed its robustness for very high dimensional data. Zhu et al [59] proposed a modified version of the SET algorithm for federated learning.…”

“…Most of the previous efforts in this direction show the effectiveness of DST in outperforming dense neural networks with high sparsity levels in supervised classification tasks [37,40,7]. Recently, DST showed its success in other domains such as text classification [32], feature selection [3], continual lifelong learning [48], and federated learning [59].…”