Training Deep Spiking Convolutional Neural Networks With STDP-Based Unsupervised Pre-training Followed by Supervised Fine-Tuning

Lee, Chankyu; Panda, Priyadarshini; Srinivasan, Gopalakrishnan; Roy, Kaushik

doi:10.3389/fnins.2018.00435

Cited by 165 publications

(114 citation statements)

References 33 publications

Supporting

Mentioning

112

Contrasting

Unclassified

Order By: Relevance

“…Current SNN models for pattern recognition can be generally categorized into three classes: that is, indirect training [12,13,14,15,16,17,18,19], direct SL training with BP [11,26,20,21,22,23,53], and plasticity-based unsupervised training with supervised modules [54,24,25]. for optimal initial weights and then used current-based BP to re-train all-layer weights in a supervised way [53]; however, this also resulted in the model being bio-implausible due to the use of the BP algorithm.…”

Section: Comparison With Other Snn Modelsmentioning

confidence: 99%

A biologically plausible supervised learning method for spiking neural networks using the symmetric STDP rule

et al. 2020

View full text Add to dashboard Cite

Spiking neural networks (SNNs) possess energy-efficient potential due to eventbased computation. However, supervised training of SNNs remains a challenge as spike activities are non-differentiable. Previous SNNs training methods can be generally categorized into two basic classes, i.e., backpropagation-like training methods and plasticity-based learning methods. The former methods are dependent on energy-inefficient real-valued computation and non-local transmission, as also required in artificial neural networks (ANNs), whereas the latter are either considered to be biologically implausible or exhibit poor performance.Hence, biologically plausible (bio-plausible) high-performance supervised learning (SL) methods for SNNs remain deficient. In this paper, we proposed a novel bio-plausible SNN model for SL based on the symmetric spike-timing dependent plasticity (sym-STDP) rule found in neuroscience. By combining the sym-STDP rule with bio-plausible synaptic scaling and intrinsic plasticity of the dynamic threshold, our SNN model implemented SL well and achieved good performance in the benchmark recognition task (MNIST dataset). To reveal the underlying mechanism of our SL model, we visualized both layer-based * These two authors contributed equally to this work.activities and synaptic weights using the t-distributed stochastic neighbor embedding (t-SNE) method after training and found that they were well clustered, thereby demonstrating excellent classification ability. Furthermore, to verify the robustness of our model, we trained it on another more realistic dataset (Fashion-MNIST), which also showed good performance. As the learning rules were bio-plausible and based purely on local spike events, our model could be easily applied to neuromorphic hardware for online training and may be helpful for understanding SL information processing at the synaptic level in biological neural systems.

show abstract

Section: Comparison With Other Snn Modelsmentioning

confidence: 99%

A biologically plausible supervised learning method for spiking neural networks using the symmetric STDP rule

et al. 2020

View full text Add to dashboard Cite

show abstract

“…However, due to discontinuous nature of communication in SNNs, direct implementing of the algorithm in DSNNs is still a challenge but there are some works trying to overcome the challenge by finding learning rules that can achieve high performance as observed in the brains [37][38][39] or by treating membrane potential of neurons as signals that shows discontinuities at spike times as noise [40].…”

Section: Discussionmentioning

confidence: 99%

Toward One-Shot Learning in Neuroscience-Inspired Deep Spiking Neural Networks

Faghihi

Molhem

Moustafa

2019

Preprint

View full text Add to dashboard Cite

Conventional deep neural networks capture essential information processing stages in perception. Deep neural networks often require very large volume of training examples, whereas children can learn concepts such as hand-written digits with few examples. The goal of this project is to develop a deep spiking neural network that can learn from few training trials. Using known neuronal mechanisms, a spiking neural network model is developed and trained to recognize hand-written digits with presenting one to four training examples for each digit taken from the MNIST database. The model detects and learns geometric features of the images from MNIST database. In this work, a novel biological back-propagation based learning rule is developed and used to a train the network to detect basic features of different digits. For this purpose, randomly initialized synaptic weights between the layers are being updated. By using a neuroscience inspired mechanism named 'synaptic pruning' and a predefined threshold, some of the synapses through the training are deleted. Hence, information channels are constructed that are highly specific for each digit as matrix of synaptic connections between two layers of spiking neural networks. These connection matrixes named 'information channels' are used in the test phase to assign a digit class to each test image. As similar to humans' abilities to learn from small training trials, the developed spiking neural network needs a very small dataset for training, compared to conventional deep learning methods checked on MNIST dataset.

show abstract

“…Note that N miss + N hit = N , if the decision of the network is based on the maximum potential of the network, if the decision of the network is based on the early spike N miss + N hit ≤ N because there might be no spikes for some inputs. Others have proposed error back propagation through all layers in spiking networks [18] [26], but this work focuses on using classifiers like a simple two layer back propagation, SVM, R-STDP.…”

Section: Reward Modulated Stdpmentioning

confidence: 99%

Feature Extraction using Spiking Convolutional Neural Networks

Vaila

Chiasson

Saxena

2019

Proceedings of the International Conference on Neuromorphic Systems

View full text Add to dashboard Cite

Spiking neural networks are biologically plausible counterparts of the artificial neural networks, artificial neural networks are usually trained with stochastic gradient descent and spiking neural networks are trained with spike timing dependant plasticity. Training deep convolutional neural networks is a memory and power intensive job. Spiking networks could potentially help in reducing the power usage. There is a large pool of tools for one to chose to train artificial neural networks of any size, on the other hand all the available tools to simulate spiking neural networks are geared towards computational neuroscience applications and they are not suitable for real life applications. In this work we focus on implementing a spiking CNN using Tensorflow to examine behaviour of the network and study catastrophic forgetting in the spiking CNN and weight initialization problem in R-STDP using MNIST data set. We also report classification accuracies that are achieved using N-MNIST and MNIST data sets. CCS CONCEPTS • Computing methodologies → Machine learning; Machine learning approaches; Bio-inspired approaches;

show abstract

Training Deep Spiking Convolutional Neural Networks With STDP-Based Unsupervised Pre-training Followed by Supervised Fine-Tuning

Cited by 165 publications

References 33 publications

A biologically plausible supervised learning method for spiking neural networks using the symmetric STDP rule

A biologically plausible supervised learning method for spiking neural networks using the symmetric STDP rule

Toward One-Shot Learning in Neuroscience-Inspired Deep Spiking Neural Networks

Feature Extraction using Spiking Convolutional Neural Networks

Contact Info

Product

Resources

About