A brain-inspired algorithm for training highly sparse neural networks

Abstract

Sparse neural networks attract increasing interest as they exhibit comparable performance to their dense counterparts while being computationally efficient. Pruning the dense neural networks is among the most widely used methods to obtain a sparse neural network. Driven by the high training cost of such methods that can be unaffordable for a low-resource device, training sparse neural networks sparsely from scratch has recently gained attention. However, existing sparse training algorithms suffer from various issues, including poor performance in high sparsity scenarios, computing dense gradient information during training, or pure random topology search. In this paper, inspired by the evolution of the biological brain and the Hebbian learning theory, we present a new sparse training approach that evolves sparse neural networks according to the behavior of neurons in the network. Concretely, by exploiting the cosine similarity metric to measure the importance of the connections, our proposed method, “Cosine similarity-based and random topology exploration (CTRE)”, evolves the topology of sparse neural networks by adding the most important connections to the network without calculating dense gradient in the backward. We carried out different experiments on eight datasets, including tabular, image, and text datasets, and demonstrate that our proposed method outperforms several state-of-the-art sparse training algorithms in extremely sparse neural networks by a large gap. The implementation code is available on Github.

Correlation Based Node Partitioning to Sparse Multilayer Perceptron

“…It considers statistics of the input and hidden node outputs to sparsify a network [23]. The Hebbian learning inspired a number of correlation-based pruning algorithms [23], [37], [41], [42]. If two neurons simultaneously fire (high correlation between their firing rates), the strength of their synaptic connection is increased [43].…”

“…Then, it removes connections between weakly correlated neurons and keeps ones with strong correlations [41]. The same idea is also tested by using the cosine similarity metric [42]. Weight matrices are sparsified starting from the last layer to the previous layers in sequence [41].…”

Correlation Based Node Partitioning to Sparse Multilayer Perceptron

“…It considers statistics of the input and hidden node outputs to sparsify a network [23]. The Hebbian learning inspired a number of correlation-based pruning algorithms [23], [37], [41], [42]. If two neurons simultaneously fire (high correlation between their firing rates), the strength of their synaptic connection is increased [43].…”

Integrating genetic markers and adiabatic quantum machine learning to improve disease resistance-based marker assisted plant selection

“…Feature selection helps avoid overfitting by removing such features (Vlasic et al, 2023). Enhances Interpretability: By selecting meaningful features, the model becomes more interpretable, allowing users to understand how the predictions are made (Atashgahi et al, 2023). Identifies Correlated Features: Feature selection identifies correlated features and removes them, reducing redundancy in the dataset (Das et al, 2022).…”