Opponent learning with different representations in the cortico-basal ganglia circuits

Abstract

While positive reward prediction errors (RPEs) and negative RPEs have equal impacts in the standard reinforcement learning, the brain appears to have distinct neural pathways for learning mainly from either positive or negative feedbacks, such as the direct and indirect pathways of the basal ganglia (BG). Given that distinct pathways may unevenly receive inputs from different neural populations and/or regions, how states or actions are represented can differ between the pathways. We explored whether combined use of different representations, coupled with different learning rates from positive and negative RPEs, has computational benefits. We considered an agent equipped with two learning systems, each of which adopted individual representation (IR) or successor representation (SR) of states. With varying the combination of IR or SR and also the learning rates from positive and negative RPEs in each system, we examined how the agent performed in a certain dynamic reward environment. We found that combination of SR-based system learning mainly from positive RPEs and IR-based system learning mainly from negative RPEs outperformed the other combinations, including IR-only or SR-only cases and the cases where the two systems had the same ratios of positive- and negative-RPE-based learning rates. In the best combination case, both systems show activities of comparable magnitudes with opposite signs, consistent with suggested profiles of BG pathways. These results suggest that particularly combining different representations with appetitive and aversive learning could be an effective learning strategy in a certain dynamic reward environment, and it might actually be implemented in the cortico-BG circuits.

Opponent Learning with Different Representations in the Cortico-Basal Ganglia Pathways Can Develop Obsession-Compulsion Cycle

“…underlies obsession-compulsion. Moreover, given that such a combination of opponent SR+IR systems actually performs well in certain dynamic environments and its implementation in the cortico-basal ganglia pathways appears consistent with various findings (Morita et al, 2022), this possibility can further explain why human is prone to OCD and even healthy people tended to show shorter memory trace for negative PEs in the previous study (Sakai et al, 2022). Besides, considering a recent finding that silencing of dorsal raphe serotonin neurons disrupts model-based control in outcome devaluation tasks (Ohmura et al, 2021), our model is in line with one of the suggested mechanisms of the effectiveness of selective serotonin reuptake inhibitors (SSRIs) for OCD that SSRIs re-engage the brain system for model-based control (Robbins et al, 2019).…”

“…These results reconcile the recent suggestion of memory trace imbalance in OCD (Sakai et al, 2022) with the long-standing suggestion that OCD is associated with impairment of model-based control (Gillan et al, 2016; Robbins et al, 2019), raising a possibility that opponent learning in modelbased (SR-based) and model-free (IR-based) controls underlies obsession-compulsion. Moreover, given that such a combination of opponent SR+IR systems actually performs well in certain dynamic environments and its implementation in the cortico-basal ganglia pathways appears consistent with various physiological and anatomical findings (Morita et al, 2022), this possibility can further explain why human is prone to OCD and even healthy people tended to show shorter memory trace for negative PEs in the previous study (Sakai et al, 2022).…”

“…Agent having a coupled SR-based system and IR-based system, adopted from the previous study (Morita et al, 2022). Each system develops the system-specific value of each action, and the average of the two system-specific values, named the integrated action value, is used for soft-max action selection and calculation of SARSA-type TD-RPEs.…”

“…As described in the Introduction, the opponent combination of appetitive SR-and aversive IR-based systems has recently been shown to perform well in certain dynamic environments [28].…”

“…We adopted the model agent developed by the recent work (Morita et al, 2022) that have coupled SR- and IR-based systems (Figure 2), with a modification that SR of actions (c.f., (Russek et al, 2017)) rather than SR of states was used in the present study because there were only two states in the environmental model. Each system developed the system-specific value of each action, and their average, referred to as the (integrated) action value, was used for action selection and TD-RPE calculation.…”