A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning

Abstract

Convergent evidence suggests that the basal ganglia support reinforcement learning by adjusting action values according to reward prediction errors. However, adaptive behavior in stochastic environments requires the consideration of uncertainty to dynamically adjust the learning rate. We consider how cholinergic tonically active interneurons (TANs) may endow the striatum with such a mechanism in computational models spanning three Marr's levels of analysis. In the neural model, TANs modulate the excitability of spiny neurons, their population response to reinforcement, and hence the effective learning rate. Long TAN pauses facilitated robustness to spurious outcomes by increasing divergence in synaptic weights between neurons coding for alternative action values, whereas short TAN pauses facilitated stochastic behavior but increased responsiveness to change-points in outcome contingencies. A feedback control system allowed TAN pauses to be dynamically modulated by uncertainty across the spiny neuron population, allowing the system to self-tune and optimize performance across stochastic environments.DOI: http://dx.doi.org/10.7554/eLife.12029.001

Dopamine waves as a mechanism for spatiotemporal credit assignment

“…While the two functions are not mutually exclusive, our data provide strong support for the second interpretation: On a trial-by-trial basis, the degree of ramping across regions was related to the latency to reward peak elicited by the wave, and the combination of ramp slope and wave magnitude was predictive of subsequent-trial behavioral adjustments. These findings accord with views that dopamine signals can have different functions during reward pursuit and outcome, which can be gated by local microcircuit elements that regulate plasticity windows (Berke, 2018;Bradfield et al, 2013;Franklin and Frank, 2015;Morris et al, 2004;Threlfell and Cragg, 2011) . Moreover, we also interpret transient and localized RPEs during reward pursuit as facilitating inference about the current task state (i.e., determining credit), whereas RPEs during reward itself facilitates reinforcement learning; a dual operation that can also be gated (Franklin and Frank, 2015;Gershman et al, 2015;Redish et al, 2007;Schoenbaum et al, 2013) .…”

“…These findings accord with views that dopamine signals can have different functions during reward pursuit and outcome, which can be gated by local microcircuit elements that regulate plasticity windows (Berke, 2018;Bradfield et al, 2013;Franklin and Frank, 2015;Morris et al, 2004;Threlfell and Cragg, 2011) . Moreover, we also interpret transient and localized RPEs during reward pursuit as facilitating inference about the current task state (i.e., determining credit), whereas RPEs during reward itself facilitates reinforcement learning; a dual operation that can also be gated (Franklin and Frank, 2015;Gershman et al, 2015;Redish et al, 2007;Schoenbaum et al, 2013) . Put together, the synthesis of our data and computational simulations imply that dopamine signals are spatio-temporally vectorized during both epochs, tailored to underlying region's computational specialty.…”

Metaplasticity as a Neural Substrate for Adaptive Learning and Choice under Uncertainty

“…In their model, the estimate of reward probability is updated only if a change is detected and if so, a new estimate of reward probability can be made depending on the location of the detected change. Interestingly, a recent modeling study has shown that increased responsiveness to change-points can be instantiated by pauses in tonically active interneurons in the striatum enabling the modulation of learning rate by reward uncertainty (Franklin and Frank, 2015). Although we did not incorporate a change-detection mechanism, such a mechanism would only improve the performance of our model (Gallistel et al, 2001; McGuire et al, 2014).…”

Regional Striatal Cholinergic Involvement in Human Behavioural Flexibility

“…This suggests a more complex mechanism in which perseveration is influenced, in part, by the learning rate from negative prediction errors (which can change due to task demand) and by resting levels of DS CHO. Indeed, Franklin et al (2015) showed that a model which takes into account cholinergic activity performs better on a reversal learning task than a model based solely on dopamine prediction error signalling (Franklin & Frank, 2015).…”