An alternative to backpropagation through time

Manneschi, Luca; Vasilaki, Eleni

doi:10.1038/s42256-020-0162-9

Cited by 13 publications

(10 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, activity-dependent compensation may provide useful techniques for machine learning. For example, we found that performance of a reservoir computing network could be improved if thresholds of individual neurons are initialized to achieve a particular activity probability given the distribution of input activities [66].…”

Section: Discussionmentioning

confidence: 99%

Compensatory variability in network parameters enhances memory performance in theDrosophilamushroom body

Abdelrahman

Vasilaki

Lin

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Neural circuits use homeostatic compensation to achieve consistent behaviour despite variability in underlying intrinsic and network parameters. However, it remains unclear how compensation regulates variability across a population of the same type of neurons within an individual, and what computational benefits might result from such compensation. We address these questions in the Drosophila mushroom body, the fly's olfactory memory center. In a computational model, we show that memory performance is degraded when the mushroom body's principal neurons, Kenyon cells (KCs), vary realistically in key parameters governing their excitability, because the resulting inter-KC variability in average activity levels makes odor representations less separable. However, memory performance is rescued while maintaining realistic variability if parameters compensate for each other to equalize KC average activity. Such compensation can be achieved through both activity-dependent and activity-independent mechanisms. Finally, we show that correlations predicted by our model's compensatory mechanisms appear in the Drosophila hemibrain connectome. These findings reveal compensatory variability in the mushroom body and describe its computational benefits for associative memory.

show abstract

Section: Discussionmentioning

confidence: 99%

Compensatory variability in network parameters enhances memory performance in theDrosophilamushroom body

Abdelrahman

Vasilaki

Lin

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…As a side note, it is very difficult to imagine how a biological neural network would be able to implement backpropagation through time, and for this alternative, approaches have recently made their appearance. [121] Reservoir computing methods came up with a workaround to the problem of training recurrent networks: they do not train them but instead harness their properties. Common in the approaches of echo state networks and liquid state machines is the idea of using a randomly recurrent network with fixed connectivity, hence no need to resort to backpropagation through time.…”

Section: Future Of Neuromorphic and Bio-inspired Computing Systemsmentioning

confidence: 99%

Section: Future Of Neuromorphic and Bio‐inspired Computing Systemsmentioning

confidence: 99%

“…Could ideas from biology still add value to existing methods? A recent augmentation of the echo state networks, [ 124 ] inspired by the fruit fly brain, explores the concept of sparseness to improve learning performance of reservoirs. In brains, contrary to the typical ANNs, only few neurons fire at a time, a fact that has been linked to memory capacity.…”

Section: Future Of Neuromorphic and Bio‐inspired Computing Systemsmentioning

confidence: 99%

See 1 more Smart Citation

Memristors—From In‐Memory Computing, Deep Learning Acceleration, and Spiking Neural Networks to the Future of Neuromorphic and Bio‐Inspired Computing

Mehonić¹,

Sebastian

Rajendran

et al. 2020

Advanced Intelligent Systems

Self Cite

197

121

View full text Add to dashboard Cite

Three factors are currently driving the main developments in artificial intelligence (AI): availability of vast amounts of data, continuous growth in computing power, and algorithmic innovations. Graphics processing units (GPUs) have been demonstrated as effective co-processors for the implementation of machine learning (ML) algorithms based on deep learning (DL). Solutions based on DL and GPU implementations have led to massive improvements in many AI tasks, but have also caused an exponential increase in demand for computing power. Recent analyses show that the demand for computing power has increased by a factor of 300 000 since 2012, and the estimate is that this demand will double every 3.4 months-at a much faster rate than improvements made historically through Moore's scaling (a sevenfold improvement over the same period of time). [1] At the same time, Moore's law has been slowing down significantly for the last few years, [2] as there are strong indications that we will not be able to continue scaling down complementary metal-oxide-semiconductor (CMOS) transistors. This calls for the exploration of alternative technology roadmaps for the development of scalable and efficient AI solutions. Transistor scaling is not the only way to improve computing performance. Architectural innovations, such as GPUs, field-programmable arrays (FPGAs), and application-specific integrated circuits (ASICs), have all significantly advanced the ML field. [3] A common aspect of modern computing architectures for ML is a move away from the classical von-Neumann architecture that physically separates memory and computing. This approach yields a performance bottleneck that is often the main reason for both energy and speed inefficiency of ML implementations on conventional hardware platforms due to

show abstract

“…This approach requires the full history of the network activity to be stored, hence it is quite unlikely that the brain performs such an algorithm for learning. An alternative to the BPTT algorithm that is more biologically realistic is the e-prop learning rule (Bellec et al, 2020), for a short summary see (Manneschi and Vasilaki, 2020). Unlike BPTT, for updating the synaptic weights of the network this learning rule requires only information locally available to each synapse and neuron -an eligibility trace and a learning signal broadcasted across the network.…”

Section: Training Rsnns With E-propmentioning

confidence: 99%

Spike-based symbolic computations on bit strings and numbers

Kraišniković

Maass

Legenstein

2021

Preprint

View full text Add to dashboard Cite

The brain uses recurrent spiking neural networks for higher cognitive functions such as symbolic computations, in particular, mathematical computations. We review the current state of research on spike-based symbolic computations of this type. In addition, we present new results which show that surprisingly small spiking neural networks can perform symbolic computations on bit sequences and numbers and even learn such computations using a biologically plausible learning rule. The resulting networks operate in a rather low firing rate regime, where they could not simply emulate artificial neural networks by encoding continuous values through firing rates. Thus, we propose here a new paradigm for symbolic computation in neural networks that provides concrete hypotheses about the organization of symbolic computations in the brain. The employed spike-based network models are the basis for drastically more energy-efficient computer hardware -- neuromorphic hardware. Hence, our results can be seen as creating a bridge from symbolic artificial intelligence to energy-efficient implementation in spike-based neuromorphic hardware.

show abstract

An alternative to backpropagation through time

Cited by 13 publications

References 6 publications

Compensatory variability in network parameters enhances memory performance in theDrosophilamushroom body

Compensatory variability in network parameters enhances memory performance in theDrosophilamushroom body

Memristors—From In‐Memory Computing, Deep Learning Acceleration, and Spiking Neural Networks to the Future of Neuromorphic and Bio‐Inspired Computing

Spike-based symbolic computations on bit strings and numbers

Contact Info

Product

Resources

About