Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive.

Abstract

The striatal dopaminergic system has been implicated in reinforcement learning (RL), motor performance, and incentive motivation. Various computational models have been proposed to account for each of these effects individually, but a formal analysis of their interactions is lacking. Here we present a novel algorithmic model expanding the classical actor-critic architecture to include fundamental interactive properties of neural circuit models, incorporating both incentive and learning effects into a single theoretical framework. The standard actor is replaced by a dual opponent actor system representing distinct striatal populations, which come to differentially specialize in discriminating positive and negative action values. Dopamine modulates the degree to which each actor component contributes to both learning and choice discriminations. In contrast to standard frameworks, this model simultaneously captures documented effects of dopamine on both learning and choice incentive-and their interactions-across a variety of studies, including probabilistic RL, effort-based choice, and motor skill learning.Keywords: dopamine, striatum, reinforcement learning, choice incentive, computational model Dopamine plays a crucial role in human and animal cognition, substantially influencing a diversity of processes including reinforcement learning, motivation, incentive, working memory, and effort. Dopaminergic neurons in the substantial nigra and ventral tegmental area project to a very wide set of subcortical and cortical areas, with strongest innervation in the basal ganglia (BG), specifically in the ventral and dorsal striatum. Dysregulation of dopamine is present in a wide array of mental illnesses such as Parkinson's disease, attention-deficit/hyperactivity disorder (ADHD), schizophrenia, and Tourette's syndrome and is a central pharmaceutical target used to treat symptoms across these and numerous other pathologies.Although considerable progress has been made in our understanding of its various distinct roles, there remain fundamental debates concerning its precise mechanisms and functions, especially regarding their integration and interactions. In reward-based decision making in particular, two largely separate traditions have studied the reinforcement learning and the incentive theories of dopamine (Berridge, 2007). Despite solid evidence for both theories, theoretical and empirical studies tend to favor and focus on one or the other interpretation, with little attempt to unify them or to study their interaction. Here we develop an explicit computational analysis of the dual role of striatal dopamine in modulation of incentive motivation (affecting choice), reinforcement learning, and how these processes interact. This endeavor allows us not only to account for both types of findings alone but also those that could not be explained by either theory in isolation. RL Theory of DopamineOne widely accepted theory of dopamine function relates to its role in model-free reinforcement learning (RL). Specifically, phasic f...

Associations between Electrophysiological Evidence of Reward and Punishment-Based Learning and Psychotic Experiences and Social Anhedonia in At-Risk Groups

“…One interpretation of this result is that increased dopamine produces increased salience for positive feedback. This interpretation is consistent with evidence that the FRN is influenced by the salience of outcomes (eg, Pfabigan et al, 2015), that increased striatal dopamine is associated with heightened sensitivity to rewards and decreased sensitivity to punishments (Collins and Frank, 2014), and that evidence of increased sensitivity to rewards is associated with increased FRN for unexpected rewards (Smillie et al, 2011). The reversal learning task (RLT) developed by Cools et al (2009) involves periodic, unexpected positive and negative feedback, with increased striatal dopamine (either baseline levels or due to pharmacological manipulation) associated with behavioral evidence of better learning after unexpected reward relative to unexpected punishment (Cools et al, 2009;van der Schaaf et al, 2014).…”

Role of the Indirect Pathway of the Basal Ganglia in Perceptual Decision Making

“…Indeed, an extension of the model with only the direct pathway (Lo and Wang, 2006) that incorporates the dopamine system and rewarddependent plasticity at corticostriatal projections was recently presented (Hsiao and Lo, 2013). Related existing work includes a cortex-BG model implementing optimal decision making for multiple choices (Bogacz and Gurney, 2007) and reinforcement learning (Bogacz and Larsen, 2011) and an extended actor-critic model providing a unified description of reinforcement learning and choice incentive, where the direct and indirect pathway striatal neurons play the roles of two opponent actors and exhibit reward-dependent learning (Collins and Frank, 2014).…”

Conflict acts as an implicit cost in reinforcement learning

“…Figure 5 shows the effects of individual differences in the genetic polymorphism for DARPP-32, which primarily affects striatal dopaminergic functioning associated with D1 and D2 pathways related to learning from positive and negative action values [26][27][28][29]45 . Absent any differences in overall task performance (easy contrast not significant), the presence of a DARPP-32 C allele was associated with an increased C4B bias in the conflict cost contrast (T/T N ¼ 35, T/C N ¼ 26, C/C N ¼ 22; t 82 ¼ 1.99, P ¼ 0.05), consistent with prior studies linking this allele to a bias towards reward-based learning and choice.…”

“…Dopamine bursts in the cortico-striatal D1 direct pathway underlies ability to learn from and seek reward, whereas dopamine dips in the D2-mediated indirect pathway underlies the ability to learn from and avoid punishment [22][23][24][25] . Computational models show how the striatal D1 and D2 pathways come to represent values and costs in such tasks, and that choices in reward-based tasks are best described by an opponent process whereby each choice option has a corresponding positive (D1) and negative (D2) action value 26 . The dopamine-and cyclic AMP-regulated phosphoprotein (DARPP-32) has been used as a marker for cortico-striatal plasticity, where an increasing number of T alleles predict an imbalance in learning favouring D1 relative to D2 pathways.…”