A Learning Theory for Reward-Modulated Spike-Timing-Dependent Plasticity with Application to Biofeedback

Abstract

Reward-modulated spike-timing-dependent plasticity (STDP) has recently emerged as a candidate for a learning rule that could explain how behaviorally relevant adaptive changes in complex networks of spiking neurons could be achieved in a self-organizing manner through local synaptic plasticity. However, the capabilities and limitations of this learning rule could so far only be tested through computer simulations. This article provides tools for an analytic treatment of reward-modulated STDP, which allows us to predict under which conditions reward-modulated STDP will achieve a desired learning effect. These analytical results imply that neurons can learn through reward-modulated STDP to classify not only spatial but also temporal firing patterns of presynaptic neurons. They also can learn to respond to specific presynaptic firing patterns with particular spike patterns. Finally, the resulting learning theory predicts that even difficult credit-assignment problems, where it is very hard to tell which synaptic weights should be modified in order to increase the global reward for the system, can be solved in a self-organizing manner through reward-modulated STDP. This yields an explanation for a fundamental experimental result on biofeedback in monkeys by Fetz and Baker. In this experiment monkeys were rewarded for increasing the firing rate of a particular neuron in the cortex and were able to solve this extremely difficult credit assignment problem. Our model for this experiment relies on a combination of reward-modulated STDP with variable spontaneous firing activity. Hence it also provides a possible functional explanation for trial-to-trial variability, which is characteristic for cortical networks of neurons but has no analogue in currently existing artificial computing systems. In addition our model demonstrates that reward-modulated STDP can be applied to all synapses in a large recurrent neural network without endangering the stability of the network dynamics.

Time scales of memory, learning, and plasticity

“…The abstract idea of this concept is that a complex network of calculating identities (e.g., neurons) is so diverse that each task is solved somewhere within the network (Maass et al 2002;Buonomano and Maass 2009;Maass 2010). However, one problem with this approach is the capacity, which depends sublinearly on the number of neurons (Ganguli et al 2008); another problem is the read-out of the task-specific information from the network (Maass et al 2007;Legenstein et al 2008). …”

Computational models of reinforcement learning: the role of dopamine as a reward signal

“…Thus DA does not further potentiate synaptic plasticity as modeled, but prevents depotentiation leading to a similar overall result of having specific potentiated synapses. The applicability of the reward-modulated model was further tested with computer simulations using networks of leaky integrate-and-fire (LIF) neurons by Legenstein et al (2008). The simulations showed the ability of this type of rule to predict spike times (rather than stimulus delivery times).…”

Reinforcement Learning in Cortical Networks

“…When the LTD-part in the STDP window is suppressed and the remaining R-STDP is bias-corrected, the learning speed for standard association tasks comes close to the one for gradient-based spike reinforcement (Frémaux et al, 2010). An elegant solution to solve the reward-bias problem is to assume that the internal reward signal is shaped by a temporal kernel that sums up to zero across time, R t dt = 0, and hence a positive internal reward signal must be followed or preceded by a negative one (Legenstein et al, 2008). What appears as a computational trick is reminiscent to the observed relieve from pain in fruit flies (Tanimoto et al, 2004), or the reward baseline adaptation in rodents (Schultz et al, 1997).…”