Multi-timescale reinforcement learning in the brain

Abstract

To thrive in complex environments, animals and artificial agents must learn to act adaptively to maximize fitness and rewards. Such adaptive behavior can be learned through reinforcement learning1, a class of algorithms that has been successful at training artificial agents2–6and at characterizing the firing of dopamine neurons in the midbrain7–9. In classical reinforcement learning, agents discount future rewards exponentially according to a single time scale, controlled by the discount factor. Here, we explore the presence of multiple timescales in biological reinforcement learning. We first show that reinforcement agents learning at a multitude of timescales possess distinct computational benefits. Next, we report that dopamine neurons in mice performing two behavioral tasks encode reward prediction error with a diversity of discount time constants. Our model explains the heterogeneity of temporal discounting in both cue-evoked transient responses and slower timescale fluctuations known as dopamine ramps. Crucially, the measured discount factor of individual neurons is correlated across the two tasks suggesting that it is a cell-specific property. Together, our results provide a new paradigm to understand functional heterogeneity in dopamine neurons, a mechanistic basis for the empirical observation that humans and animals use non-exponential discounts in many situations10–14, and open new avenues for the design of more efficient reinforcement learning algorithms.

The role of prospective contingency in the control of behavior and dopamine signals during associative learning

“…The transition and observation matrix, which are used to compute the probability of each state, were derived from the experimental settings, assuming a fixed probability of transition from the Wait to Pre state, modeling a growing anticipation of the next trial beginning. Using this state representation improved the quantitative accuracy of the model for a given γ versus the Cue-Context, and accurately predicted the experimental data at a value of consistent with previously reported results 46–49 (Fig 3f., Extended Data Fig. 4).…”

“…By increasing the context and thus value during the pre-Odor ITI period, the TD error at Odor A is diminished. While this produces a qualitatively correct pattern of results, it requires a temporal discount factor that is well below previously reported values 46–49 to produce the quantitatively correct pattern (Extended Data Fig. 4).…”

A feature-specific prediction error model explains dopaminergic heterogeneity

“…Though recent work has shown that DA in the tail of the striatum elicited by cue-evoked action execution decreases over time 40 , supporting the idea that action responses are modulated by predictability rather than simple motor correlates, other experiments must still be conducted in order to establish that DA neurons encode an APE fully analogous to the TD RPE. Notably, a TD prediction error transfers with learning from the predicted outcome to cues predicting it (at least depending on the time discount parameter, which might also vary between circuits 26,99,100 ). By analogy, a TD APE predicts that cues that elicit actions should produce DA signals that increase with training, and that “omission” of an action predicted by a cue (say, by introduction of a no-go signal) should yield a reduction in DA.…”