Topological Insights into Sparse Neural Networks

Liu, Shiwei; Lee, Tim van der; Yaman, Anil; Atashgahi, Zahra; Ferraro, Davide; Sokar, Ghada; Pechenizkiy, Mykola; Mocanu, Decebal Constantin

doi:10.1007/978-3-030-67664-3_17

Cited by 17 publications

(15 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This indicates that in the case of RNNs there exist many low-dimensional sub-networks that can achieve similarly low loss. This phenomenon complements the findings of Liu et al (2020c) which shows that there are numerous sparse sub-networks performing similarly well in the context of sparse MLPs.…”

Section: Analysis Of Evolutionary Trajectory Of Dynamic Sparse Trainingsupporting

confidence: 86%

Selfish Sparse RNN Training

Liu¹,

Mocanu²,

Pei³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Sparse neural networks have been widely applied to reduce the necessary resource requirements to train and deploy over-parameterized deep neural networks. For inference acceleration, methods that induce sparsity from a pre-trained dense network (dense-to-sparse training) work effectively. Recently, dynamic sparse training (DST) has been proposed to train sparse neural networks without pre-training a dense network (sparse-to-sparse training), so that the training process can also be accelerated. However, previous sparse-to-sparse methods mainly focus on Multilayer Perceptron Networks (MLPs) and Convolutional Neural Networks (CNNs), failing to match the performance of dense-to-sparse methods in Recurrent Neural Networks (RNNs) setting. In this paper, we propose an approach to train sparse RNNs with a fixed parameter count in one single run, without compromising performance. During training, we allow RNN layers to have a non-uniform redistribution across cell gates for a better regularization. Further, we introduce SNT-ASGD, a variant of the averaged stochastic gradient optimizer, which significantly improves the performance of all sparse training methods for RNNs. Using these strategies, we achieve state-of-the-art sparse training results, better than the dense-to-sparse methods, with various types of RNNs on Penn TreeBank and Wikitext-2 datasets.

show abstract

Section: Analysis Of Evolutionary Trajectory Of Dynamic Sparse Trainingsupporting

confidence: 86%

Selfish Sparse RNN Training

Liu¹,

Mocanu²,

Pei³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Reference [35] proposed a method to measure the distance between sparse connectivities obtained with dynamic sparse training from the perspective of graph theory. They empirically show that there are many different sparse connectivities achieving equally good performance.…”

Section: Sparse Neural Network Throughout Trainingmentioning

confidence: 99%

Efficient and effective training of sparse recurrent neural networks

Liu

Ni'mah

Menkovski

et al. 2021

Neural Comput & Applic

Self Cite

View full text Add to dashboard Cite

Recurrent neural networks (RNNs) have achieved state-of-the-art performances on various applications. However, RNNs are prone to be memory-bandwidth limited in practical applications and need both long periods of training and inference time. The aforementioned problems are at odds with training and deploying RNNs on resource-limited devices where the memory and floating-point operations (FLOPs) budget are strictly constrained. To address this problem, conventional model compression techniques usually focus on reducing inference costs, operating on a costly pre-trained model. Recently, dynamic sparse training has been proposed to accelerate the training process by directly training sparse neural networks from scratch. However, previous sparse training techniques are mainly designed for convolutional neural networks and multi-layer perceptron. In this paper, we introduce a method to train intrinsically sparse RNN models with a fixed number of parameters and floating-point operations (FLOPs) during training. We demonstrate state-of-the-art sparse performance with long short-term memory and recurrent highway networks on widely used tasks, language modeling, and text classification. We simply use the results to advocate that, contrary to the general belief that training a sparse neural network from scratch leads to worse performance than dense networks, sparse training with adaptive connectivity can usually achieve better performance than dense models for RNNs.

show abstract

“…initialized from a standard normal distribution. The random addition of new connections do not have a high risk of not finding good sparse connectivity at the end of the training process because it has been shown in (Liu et al 2020) that sparse training can unveil a vast number of very different sparse connectivity local optima which achieve very similar performance.…”

Section: Training Proceduresmentioning

confidence: 99%

Quick and robust feature selection: the strength of energy-efficient sparse training for autoencoders

et al. 2021

Self Cite

View full text Add to dashboard Cite

Major complications arise from the recent increase in the amount of high-dimensional data, including high computational costs and memory requirements. Feature selection, which identifies the most relevant and informative attributes of a dataset, has been introduced as a solution to this problem. Most of the existing feature selection methods are computationally inefficient; inefficient algorithms lead to high energy consumption, which is not desirable for devices with limited computational and energy resources. In this paper, a novel and flexible method for unsupervised feature selection is proposed. This method, named QuickSelection (The code is available at: https://github.com/zahraatashgahi/QuickSelection), introduces the strength of the neuron in sparse neural networks as a criterion to measure the feature importance. This criterion, blended with sparsely connected denoising autoencoders trained with the sparse evolutionary training procedure, derives the importance of all input features simultaneously. We implement QuickSelection in a purely sparse manner as opposed to the typical approach of using a binary mask over connections to simulate sparsity. It results in a considerable speed increase and memory reduction. When tested on several benchmark datasets, including five low-dimensional and three high-dimensional datasets, the proposed method is able to achieve the best trade-off of classification and clustering accuracy, running time, and maximum memory usage, among widely used approaches for feature selection. Besides, our proposed method requires the least amount of energy among the state-of-the-art autoencoder-based feature selection methods.

show abstract

Topological Insights into Sparse Neural Networks

Cited by 17 publications

References 10 publications

Selfish Sparse RNN Training

Selfish Sparse RNN Training

Efficient and effective training of sparse recurrent neural networks

Quick and robust feature selection: the strength of energy-efficient sparse training for autoencoders

Contact Info

Product

Resources

About