A Learning Framework for Winner-Take-All Networks with Stochastic Synapses

Mostafa, Hesham; Cauwenberghs, Gert

doi:10.1162/neco_a_01080

Cited by 14 publications

(11 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A useful precedent is given by Mostafa and Cauwenberghs (2018), in which they approximate the SoftMax function with the probability that each activation in the receiving layer, Gaussian distributed at the limit of input layer size, is higher than the activation with the largest mean value. We would therefore need to solve for dropout probabilities such that the softmax approximation for each receiving neuron i is as close as possible to normalized…”

Section: Appendix A: Induction Proof For the Primary Resultsmentioning

confidence: 99%

“…Variational inference refers to the use of greatly simplified, approximate distributions to draw inferences from complex models. Variational neural networks were pioneered with the Boltzmann machine (Hinton, 2007), Helmholtz machine (Dayan et al, 1995), and their later generalization, the variational autoencoder (VAE, Kingma and Welling, 2014) and have since been further generalized to networks of many kinds (Zhang et al, 2019), including networks that imitate neurobiology (Mostafa and Cauwenberghs, 2018;Neftci et al, 2016). In these models, sampling from the variational distributions is used to approximate an intractable integral in the objective function, to learn regularized solutions, and to generate plausible out-of-sample realizations from the learned, latent distributions.…”

Section: Probabilistic Reasoningmentioning

confidence: 99%

“…The widely-used dropout technique in deep learning can be seen as a form of variational inference and sampling (Srivastava et al, 2014;Gal and Ghahramani, 2016) with direct analogy to the random failure of synapses in the brain. This link has led to biologically-motivated models of variational deep learning that use network weight dropout to simulate synaptic failure (Mostafa and Cauwenberghs, 2018;Wan et al, 2013;Neftci et al, 2016).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Locally Learned Synaptic Dropout for Complete Bayesian Inference

McKee¹,

Crandell²,

Chaudhuri³

et al. 2021

Preprint

View full text Add to dashboard Cite

The Bayesian brain hypothesis postulates that the brain accurately operates on statistical distributions according to Bayes' theorem. The random failure of presynaptic vesicles to release neurotransmitters may allow the brain to sample from posterior distributions of network parameters, interpreted as epistemic uncertainty. It has not been shown previously how random failures might allow networks to sample from observed distributions, also known as aleatoric or residual uncertainty. Sampling from both distributions enables probabilistic inference, efficient search, and creative or generative problem solving. We demonstrate that under a population-code based interpretation of neural activity, both types of distribution can be represented and sampled with synaptic failure alone. We first define a biologically constrained neural network and sampling scheme based on synaptic failure and lateral inhibition. Within this framework, we derive dropout based epistemic uncertainty, then prove an analytic mapping from synaptic efficacy to release probability that allows networks to sample from arbitrary, learned distributions represented by a receiving layer. Second, our result leads to a local learning rule by which synapses adapt their release probabilities. Our result demonstrates complete Bayesian inference, related to the variational learning method of dropout, in a biologically constrained network using only locally-learned synaptic failure rates.

show abstract

Section: Appendix A: Induction Proof For the Primary Resultsmentioning

confidence: 99%

Section: Probabilistic Reasoningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Locally Learned Synaptic Dropout for Complete Bayesian Inference

McKee¹,

Crandell²,

Chaudhuri³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Some examples can be found in [9]. We can find models using soft non-linearities [16], probabilistic models [17] or latency-based networks [18].…”

Section: Direct Trainingmentioning

confidence: 99%

Keys to Accurate Feature Extraction Using Residual Spiking Neural Networks

Vicente-Sola,

Manna,

Kirkland

et al. 2021

Preprint

View full text Add to dashboard Cite

Spiking neural networks (SNNs) have become an interesting alternative to conventional artificial neural networks (ANN) thanks to their temporal processing capabilities and their low-SWaP (Size, Weight, and Power) and energy efficient implementations in neuromorphic hardware. However the challenges involved in training SNNs have limited their performance in terms of accuracy and thus their applications. Improving learning algorithms and neural architectures for a more accurate feature extraction is therefore one of the current priorities in SNN research. In this paper we present a study on the key components of modern spiking architectures. We empirically compare different techniques in image classification datasets taken from the best performing networks. We design a spiking version of the successful residual network (ResNet) architecture and test different components and training strategies on it. Our results provide a state of the art guide to SNN design, which allows to make informed choices when trying to build the optimal visual feature extractor. Finally, our network outperforms previous SNN architectures in and CIFAR-100 (74.5%) datasets and matches the state of the art in DVS-CIFAR10 (71.3%), with less parameters than the previous state of the art and without the need for ANN-SNN conversion. Code available at https://github. com/VicenteAlex/Spiking_ResNet

show abstract

“…In addition, each neuron is allowed to fire only once. Neurons belonging to the same feature map share the same input synaptic weights, and compete with each other to update their shared input synaptic weights according to a WTA mechanism [72]. The neuron that fires the earliest is the winner which is qualified to modify their shared weights according to STDP learning rule or imbalanced R-STDP learning rule.…”

Section: Convolutional Layermentioning

confidence: 99%

An Imbalanced R-STDP Learning Rule in Spiking Neural Networks for Medical Image Classification

Zhou

Ren

2020

IEEE Access

View full text Add to dashboard Cite

Spiking neural networks (SNNs) have the advantages of inherent power-efficiency, biological plausibility and good image recognition performance. They are good candidates for medical image classification especially when the labeled training data are limited. In medical image classification, one of the major challenges is the highly class imbalanced problem which causes deep learning networks to bias towards the majority class and poorly recognizes the minority class. Despite that there are some methods for addressing this problem, very few algorithm-level methods exist for SNNs. In this work, we propose an imbalanced reward-modulation spike-timing-dependent plasticity (R-STDP) learning rule for SNNs to solve the medical image class imbalanced problem. We introduce an imbalanced reward coefficient for the R-STDP learning rule to set the reward from the minority class to be higher than that of the majority class, and this reward coefficient can help to set the class-dependent rewards according to the data statistic of the training dataset. Experiment results on three benchmark datasets with imbalanced splits show that our method significantly improves the performance than that of the baseline SNNs and outperforms the compared state-of-the-art methods addressing medical image class imbalanced problem including datalevel and algorithm-level methods. Moreover, our method achieves excellent classification performance on the imbalanced medical dataset ISIC-2018. The results show that the proposed method can well help SNNs in classifying imbalanced medical image datasets. Besides, our proposed method can obtain high sensitivity to disease class by adjusting the reward coefficient, which is very useful for identifying disease samples in medical diagnostic tasks.INDEX TERMS Class imbalance, spiking neural networks (SNNs), reward-modulated spike-timingdependent plasticity (R-STDP), medical image.

show abstract

A Learning Framework for Winner-Take-All Networks with Stochastic Synapses

Cited by 14 publications

References 56 publications

Locally Learned Synaptic Dropout for Complete Bayesian Inference

Locally Learned Synaptic Dropout for Complete Bayesian Inference

Keys to Accurate Feature Extraction Using Residual Spiking Neural Networks

An Imbalanced R-STDP Learning Rule in Spiking Neural Networks for Medical Image Classification

Contact Info

Product

Resources

About