Efficient Computation Based on Stochastic Spikes

Ernst, Udo; Rotermund, David; Pawelzik, Klaus

doi:10.1162/neco.2007.19.5.1313

Cited by 10 publications

(46 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It was previously shown how to iteratively estimate priors in a spiking neural model, 30 but our work has the goal of combining learning with inference. Our architecture is somewhat tricky.…”

Section: B Neural Implementation Of Inferencementioning

confidence: 99%

Neuronal integration of dynamic sources: Bayesian learning and Bayesian inference

Siegelmann

Holzman

2010

Chaos: An Interdisciplinary Journal of Nonlinear Science

View full text Add to dashboard Cite

One of the brain's most basic functions is integrating sensory data from diverse sources. This ability causes us to question whether the neural system is computationally capable of intelligently integrating data, not only when sources have known, fixed relative dependencies but also when it must determine such relative weightings based on dynamic conditions, and then use these learned weightings to accurately infer information about the world. We suggest that the brain is, in fact, fully capable of computing this parallel task in a single network and describe a neural inspired circuit with this property. Our implementation suggests the possibility that evidence learning requires a more complex organization of the network than was previously assumed, where neurons have different specialties, whose emergence brings the desired adaptivity seen in human online inference. © 2010 American Institute of Physics. ͓doi:10.1063/1.3491237͔Our senses work in parallel passing multimodal data of the same fact or object to the brain. A fundamental question in the field of computational neuroscience is how the brain accommodates sensory data from different sources to form one holistic picture. Cue integration experiments, in which subjects experience apparently synchronized cross modal stimuli, but where one source is displaced from its counterpart, can reveal how the brain handles parallel inputs. The results suggest that the human brain computes a weighted average over the different cues, fitting neatly with Bayes' probability theory, where each piece of information is weighted by the amount, on which the brain relies on the channeling sense. Studies now suggest that this relative reliability must also take into account changing conditions, such as varying lighting for vision, environmental noise levels for audio, etc. Using fixed reliabilities, as was previously modeled, skews decision making and contradicts both optimality and human studies. A question arises as to whether the previous Bayes-based theory still holds for cue integration given the need of adapting the levels of reliabilities. We propose it does and introduce a neural network architecture that can both constantly learn the reliabilities of sensory data and use them in the integration of cues. While the brain does not necessarily use the method that we are proposing for our architecture and there is currently no way to test exactly how the brain makes these computations, our work provides a proof of concept of the brain's ability to handle learning and inference in parallel and within one network.

show abstract

Section: B Neural Implementation Of Inferencementioning

confidence: 99%

Neuronal integration of dynamic sources: Bayesian learning and Bayesian inference

Siegelmann

Holzman

2010

Chaos: An Interdisciplinary Journal of Nonlinear Science

View full text Add to dashboard Cite

show abstract

“…low power consumption, fast inference, event-driven information processing, and massive parallelization) ( [10]). [11] presented a type of shallow neuronal network that is based on non-negative generative models ( [12,13]). Compared to other models with spiking neurons, this Spike-By-Spike (SbS) network model provides a type of network that requires relatively low additional computational requirements for using spikes as a mean for transmitting information to other neurons.…”

Section: Introductionmentioning

confidence: 99%

“…For a network consisting of one input layer and only one hidden layer, an iterative algorithm can be derived ( [11]) which has the goal to maximize the likelihood between the observed spikes, generated by the input layer, and an internal representation of the input based on hidden latent variables. The input layer is defined by an input probability distribution p µ (s) for input pattern µ to generate a (next) spike at input neuron s. For a given pattern µ in every time step t a spike is drawn at input neuron s t,µ from p µ (s).…”

Section: Introductionmentioning

confidence: 99%

Back-propagation learning in deep Spike-By-Spike networks

Rotermund

Pawelzik

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Neural networks are important building blocks in artificial intelligence applications. These technical systems rely on noiseless continuous signals in stark contrast to the discrete action potentials exchanged among the neurons in real brains. A promising approach towards bridging this gap are the Spike-by-Spike (SbS) networks, which were demonstrated to reliably process information based on few stochastic spikes given suitable synaptic weights are provided. What is missing are algorithms for finding weight sets that would optimize the output performance of acyclically connected SbS networks.Here, a learning rule for hierarchically organized SbS networks is derived. It is inspired by the error backpropagation algorithms used in feed-forward neural networks. The properties of this approach are investigated and its functionality demonstrated by simulations. In particular, a Deep Convolutional SbS network for classifying handwritten digits (MNIST) is presented. The resulting classification performance reaches values achieved by standard deep networks that have a similar structure and also use only simple gradient descent. We expect that this new SbS back-prop learning rule will provide a basis for building spiking neuronal network models for research in neuroscience and technical applications, especially when it will become combined with state-of-the-art optimization techniques and implemented on specialized computational hardware.

show abstract

“…Here we investigated the feasibility of CS with rate coding neurons. Also, we are interested in the speed-precision tradeoff of reconstructions using spike-based algorithms similar to the one we introduced previously [3]. …”

mentioning

confidence: 99%

“…Therefore the construction of sparse representations from spikes can be considered a bias favoring speed in contrast to faithfulness. In [3] we showed that learning generating matrices is possible using only spike activity. Taken together, our results underline the potential relevance of CS for understanding connectivity structures, sparseness and activity dynamics in the brain.…”

mentioning

confidence: 99%