Functional Requirements for Reward-Modulated Spike-Timing-Dependent Plasticity

Abstract

Recent experiments have shown that spike-timing-dependent plasticity is influenced by neuromodulation. We derive theoretical conditions for successful learning of reward-related behavior for a large class of learning rules where Hebbian synaptic plasticity is conditioned on a global modulatory factor signaling reward. We show that all learning rules in this class can be separated into a term that captures the covariance of neuronal firing and reward and a second term that presents the influence of unsupervised learning. The unsupervised term, which is, in general, detrimental for reward-based learning, can be suppressed if the neuromodulatory signal encodes the difference between the reward and the expected reward-but only if the expected reward is calculated for each task and stimulus separately. If several tasks are to be learned simultaneously, the nervous system needs an internal critic that is able to predict the expected reward for arbitrary stimuli. We show that, with a critic, reward-modulated spike-timing-dependent plasticity is capable of learning motor trajectories with a temporal resolution of tens of milliseconds. The relation to temporal difference learning, the relevance of block-based learning paradigms, and the limitations of learning with a critic are discussed.

Evidence for a causal inverse model in an avian cortico-basal ganglia circuit

“…If reinforcement signals (e.g., dopamine) were mediated via causal inverse models instead of being released nonspecifically as assumed in many computational models (46,47), then motor learning could be more efficient, in accordance with model-based reinforcement strategies (48). Indeed, simple reinforcement learning strategies can be enhanced with inverse models as a means to solve the structural credit assignment problem inherent in reinforcement learning (49).…”

Reinforcement learning of a spiking neural network in the task of control of an agent in a virtual discrete environment

“…При STDP вес синапса возрастает, если входной спайк пришел до выходного, и убывает, если входной спайк пришел после выходного. Экспоненци-альное забывание сенсорной истории при применении правила минимизации энтропии (4.1) приводит к модуляции изменений весов (3.3) значениями сигнала подкрепления [11] -яв-лению Modulated STDP, также наблюдаемому в биологических нейронах и исследованному в работах [37][38][39].…”

From modulated Hebbian plasticity to simple behavior learning through noise and weight saturation

“…As opposed to the algorithms proposed by Pfeiffer et al (2010), Legenstein et al (2010) and Frémaux et al (2010), the present work neither devises a learning rule for optimal weight tuning nor proposes a new reinforcement learning algorithm. In fact, while reinforcement learning by means of modulated spike-timing dependent plasticity (STDP) was demonstrated in Soula et al (2005), Florian (2007) and Frémaux et al (2010), the primary aim of this work is the exploitation of saturated weights and neural noise to achieve a simple bottom-up implementation of oper-ant reward learning.…”

“…The change in modulatory activity has in fact been suggested to regulate the alternation of exploration and exploitation in Krichmar (2008). Thus, while the dynamics of modulated Hebbian plasticity and modulated spike-timedependent plasticity (STDP) have been extensively investigated (Abbott, 1990;Montague et al, 1996;Florian, 2007;Porr and Wörgötter, 2007;Frémaux et al, 2010;Pfeiffer et al, 2010), the novelty of this work is their extension by means of saturation and noise, resulting in a simpler and more fundamental connection between local changes and higher-level simulated behavior. The fundamental properties of the new plasticity model are tested in behavioral tasks employing first a single-neuron model, and later extended to multi-neuron networks.…”

“…In fact, while reinforcement learning by means of modulated spike-timing dependent plasticity (STDP) was demonstrated in Soula et al (2005), Florian (2007) and Frémaux et al (2010), the primary aim of this work is the exploitation of saturated weights and neural noise to achieve a simple bottom-up implementation of oper-ant reward learning. Furthermore, in contrast to Pfeiffer et al (2010), the current algorithm does not require a decay function, input signal preprocessing nor winnertake-all action selection.…”