Adaptive reinforcement learning through evolving self-modifying neural networks

Schmidgall, Samuel

doi:10.1145/3377929.3389901

Cited by 7 publications

(7 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Synaptic plasticity is a powerful mechanism for unsupervised learning in neural networks, inspired by learning processes in the biological brain [1,2,3,4,5]. This process has been incorporated into spiking and artificial neural networks to enable intra-lifetime learning [8,9,10,11,12,13]. However, in this work it was shown that plastic ANNs struggle to generalize their behavior beyond the training time horizon.…”

Section: Discussionmentioning

confidence: 96%

“…In addition, these methods, as a product of not using backpropagated gradients, do not perform backpropagation through time, and hence time-dependent parameters, like synaptic plasticity, do not require immense compute time. Particularly in the context of synaptic plasticity, some situations display evolutionary algorithms outperforming gradient-based approaches in both learned performance and in training time [25,26].…”

Section: Evolutionary Strategiesmentioning

confidence: 99%

“…In the field of Artificial Intelligence (AI), the primary focus of research with ANNs has been on discovering static solutions, where the synaptic weights remain constant throughout the lifetime of the organism. Inspired by the biological brain, a rich history of work has demonstrated the design of ANNs together with synaptic plasticity [8,9,10,11,12,13], referred to as Plastic Artificial Neural Networks (PANNs). These networks have shown impressive capabilities in their ability to generalize to novel environmental circumstances, recover from limb damage, enhance memory [11,14,5].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Stable Lifelong Learning: Spiking neurons as a solution to instability in plastic neural networks

Schmidgall¹,

Hays²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Synaptic plasticity poses itself as a powerful method of self-regulated unsupervised learning in neural networks. A recent resurgence of interest has developed in utilizing Artificial Neural Networks (ANNs) together with synaptic plasticity for intra-lifetime learning. Plasticity has been shown to improve the learning capabilities of these networks in generalizing to novel environmental circumstances. However, the long-term stability of these trained networks has yet to be examined. This work demonstrates that utilizing plasticity together with ANNs leads to instability beyond the prespecified lifespan used during training. This instability can lead to the dramatic decline of reward seeking behavior, or quickly lead to reaching environment terminal states. This behavior is shown to hold consistent for several plasticity rules on two different environments across many training time-horizons: a cart-pole balancing problem and a quadrupedal locomotion problem. We present a solution to this instability through the use of spiking neurons.

show abstract

Section: Discussionmentioning

confidence: 96%

Section: Evolutionary Strategiesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Stable Lifelong Learning: Spiking neurons as a solution to instability in plastic neural networks

Schmidgall¹,

Hays²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…There are also more recent works on self-modifying NNs. Neuromodulated plasticity is a Hebbian-style self-modification (Miconi et al, 2018;Schmidgall, 2020;Najarro & Risi, 2020) which also makes use of outer products to generate a modulation term which is added to the base weights. The corresponding computations can also be interpreted as key/value/query association operations.…”

Section: Related Workmentioning

confidence: 99%

A Modern Self-Referential Weight Matrix That Learns to Modify Itself

Irie¹,

Schlag²,

Csordás³

et al. 2022

Preprint

View full text Add to dashboard Cite

The weight matrix (WM) of a neural network (NN) is its program. The programs of many traditional NNs are learned through gradient descent in some error function, then remain fixed. The WM of a self-referential NN, however, can keep rapidly modifying all of itself during runtime. In principle, such NNs can meta-learn to learn, and metameta-learn to meta-learn to learn, and so on, in the sense of recursive self-improvement. While NN architectures potentially capable of implementing such behavior have been proposed since the '90s, there have been few if any practical studies.Here we revisit such NNs, building upon recent successes of fast weight programmers and closely related linear Transformers. We propose a scalable self-referential WM (SRWM) that uses outer products and the delta update rule to modify itself. We evaluate our SRWM in supervised few-shot learning and in multi-task reinforcement learning with procedurally generated game environments. Our experiments demonstrate both practical applicability and competitive performance of the proposed SRWM. Our code is public † .

show abstract

“…This is primarily due to the use of spikes for information transmission, which does not naturally lend itself toward being used with backpropagation. To circumvent this challenge, a wide variety of learning algorithms have been proposed including Spike-Timing Dependent Plasticity (STDP) (Masquelier et al, 2009;Bengio et al, 2017;Kheradpisheh et al, 2018;Mozafari et al, 2018), ANN to SNN conversion methods (Diehl et al, 2015;Rueckauer et al, 2017;Hu et al, 2018), Eligibility Traces (Bellec et al, 2020), and Evolutionary Strategies (Pavlidis et al, 2005;Carlson et al, 2014;Eskandari et al, 2016;Schmidgall, 2020). However, a separate body of literature enables the use of backpropagation directly with SNNs typically through the use of surrogate gradients (Bohte et al, 2002;Sporea and Grüning, 2012;Lee et al, 2016;Shrestha and Orchard, 2018).…”

Section: Introduction and Related Workmentioning

confidence: 99%

SpikePropamine: Differentiable Plasticity in Spiking Neural Networks

et al. 2021

Self Cite

View full text Add to dashboard Cite

The adaptive changes in synaptic efficacy that occur between spiking neurons have been demonstrated to play a critical role in learning for biological neural networks. Despite this source of inspiration, many learning focused applications using Spiking Neural Networks (SNNs) retain static synaptic connections, preventing additional learning after the initial training period. Here, we introduce a framework for simultaneously learning the underlying fixed-weights and the rules governing the dynamics of synaptic plasticity and neuromodulated synaptic plasticity in SNNs through gradient descent. We further demonstrate the capabilities of this framework on a series of challenging benchmarks, learning the parameters of several plasticity rules including BCM, Oja's, and their respective set of neuromodulatory variants. The experimental results display that SNNs augmented with differentiable plasticity are sufficient for solving a set of challenging temporal learning tasks that a traditional SNN fails to solve, even in the presence of significant noise. These networks are also shown to be capable of producing locomotion on a high-dimensional robotic learning task, where near-minimal degradation in performance is observed in the presence of novel conditions not seen during the initial training period.

show abstract

Adaptive reinforcement learning through evolving self-modifying neural networks

Cited by 7 publications

References 3 publications

Stable Lifelong Learning: Spiking neurons as a solution to instability in plastic neural networks

Stable Lifelong Learning: Spiking neurons as a solution to instability in plastic neural networks

A Modern Self-Referential Weight Matrix That Learns to Modify Itself

SpikePropamine: Differentiable Plasticity in Spiking Neural Networks

Contact Info

Product

Resources

About