What is dopamine doing in model-based reinforcement learning?

Abstract

Experiments have implicated dopamine in model-based reinforcement learning (RL). These findings are unexpected as dopamine is thought to encode a reward prediction error (RPE), which is the key teaching signal in model-free RL. Here we examine two possible accounts for dopamine's involvement in model-based RL: the first that dopamine neurons carry a prediction error used to update a type of predictive state representation called a successor representation, the second that two well established aspects of dopaminergic activity, RPEs and surprise signals, can together explain dopamine's involvement in model-based RL.

Internally Triggered Experiences of Hedonic Valence in Nonhuman Animals: Cognitive and Welfare Considerations

“…In particular, Takahashi et al (2017) showed that dopamine carries information about the sensory features of expected rewards in rats. This evidence is consistent with a view on which SWR-based re- and preplay events have a role in both model-free and model-based value computation (e.g., Akam & Walton, 2021). Indeed, Mattar and Daw (2018) proposed that the various functional perspectives on hippocampal re- and preplay could be unified by a view that assumes that both phenomena serve the computation of long-run action values.…”

Devaluing memories of reward: A case for dopamine

“…To achieve this, a subset of dopamine cells are required to undergo computational processes that are more elaborate than traditional models of dopamine function predict and may in part reflect dopamine's role in sensory prediction error 12,33 . Future studies targeting the nature of any underlying circuitry encoding these detailed reinforcement signals via dopamine should include the extent to which our findings relate to proposed roles in model-based encoding 12 , or signaling of surprise from afferent sensory systems 58 that are relayed to higher-order cortical sites 32 . Alternatively, given our approach intersects appetition and aversion, it is possible that we are engaging a heterogeneous population of dopamine cells that include those that respond to high intensity sensory stimuli and are aversive when stimulated 59 .…”

Dissociable feedback valence effects on frontal midline theta during reward gain versus threat avoidance learning

“…This suggestion is in line with the idea of a stronger utilization of more explicit (model‐based) expectations in the present study, masking expectations which are based on trial‐by‐trial value associations as derived by model‐free Q learning (Botvinick & Weinstein, 2014; Daw et al, 2011; Wang, Kurth‐Nelson, et al, 2018). Besides the task demands, it might also be that assessing self‐reported expectations within each trial further contributed to participants' overall use of more explicit predictions (Akam & Walton, 2021; Doll et al, 2009; Smittenaar et al, 2013). Overall, since negative PEs were not indicative of the need for behavioral adjustment, it may be that FMθ was less affected by trial‐by‐trial predictions but more by implicit learning about the reversals, decoupling the often‐replicated association between PEs and FMθ (Cooper et al, 2019; Kaufman et al, 2010; Pinner & Cavanagh, 2017; Reber, 2013).…”