Learning to Express Reward Prediction Error-like Dopaminergic Activity Requires Plastic Representations of Time

Abstract

Dopamine (DA) releasing neurons in the midbrain learn response patterns that represent reward prediction error (RPE). Typically, models proposing a mechanistic explanation for how dopamine neurons learn to exhibit RPE are based on temporal difference (TD) learning, a machine learning algorithm. However, mechanistic models motivated by TD learning face two significant hurdles. First, TD-based models typically require rather unrealistic components, such as long and robust temporal chains of feature-specific neurons that tile, a priori, each interval from a given stimulus to a given reward. Secondly, various predictions of TD clash with experimental observations of how RPE evolves over learning. Here, we present a biophysically plausible plastic network model of spiking neurons, that learns RPEs and can replicate results observed in multiple experiments. This model, coined FLEX (Flexibly Learned Errors in Expected Reward), learns feature-specific representations of time, allowing for neural representations of stimuli to adjust their timing and relation to rewards in an online manner. Following learning, model dopamine neurons in FLEX report a distribution of response types, as observed experimentally and as used in machine learning. Dopamine neuron firing in our model reflects an RPE before and after learning but not necessarily during learning, allowing the model to reconcile seemingly inconsistent experiments and make unique predictions that contrast those of TD.

Learning produces a hippocampal cognitive map in the form of an orthogonalized state machine

“…These mechanisms may work in tandem with Hebbian plasticity to construct cognitive maps and/or they may be more involved in refining behavior policy and other task-specific functions by selectively routing information from the established cognitive maps to other brain regions mediating behavioral policies. Our data indicate that task representations and behavioral policies based upon them are formed in lockstep, as suggested by previous theory 75 . A likely candidate mechanism for the contribution of synaptic plasticity during feedback is behavioral time scale synaptic plasticity (BTSP) 76 .…”

Dopamine Release Plateau and Outcome Signals in Dorsal Striatum Contrast with Classic Reinforcement Learning Formulations

“…It was only for outcome signals recorded in lateral sites that we could detect systematic changes related to changing probabilities of reward. Despite these uncertainties, our observations introduce evidence for dopamine plateau responses as learning-related features to add to transient and ramping responses formerly reported, and raise new questions about RPE encoding by the striatum during learning 39 .…”

“…For example, some cholinergic inputs, some likely from these interneurons, generate action potentials in intrastriatal dopamine fibers far from their cell bodies 16 . Further, oscillatory local field potentials can accompany and even modulate activity 39,[51][52][53] . We did not monitor this activity.…”

“…The dopamine release signals can be principally related to negative as well as positive reinforcement 5,[27][28][29] or to non-reward parameters of movement 21,30,31 , can occur as prolonged ramping signals 32 , and can be compartmentally selective for striosome and matrix compartments of the striatum 18,[33][34][35][36] . Especially for the nucleus accumbens, but also for the dorsal striatum, the relation of the release patterns to RPE-TD learning algorithms has been strongly questioned [37][38][39] and strongly defended 10,23,30,37,[39][40][41][42][43][44][45][46][47] .…”