Opponent Learning with Different Representations in the Cortico-Basal Ganglia Circuits

Abstract

The direct and indirect pathways of the basal ganglia (BG) have been suggested to learn mainly from positive and negative feedbacks, respectively. Since these pathways unevenly receive inputs from different cortical neuron types and/or regions, they may preferentially use different state/action representations. We explored whether such a combined use of different representations, coupled with different learning rates from positive and negative reward prediction errors (RPEs), has computational benefits. We modeled animal as an agent equipped with two learning systems, each of which adopted individual representation (IR) or successor representation (SR) of states. With varying the combination of IR or SR and also the learning rates from positive and negative RPEs in each system, we examined how the agent performed in a dynamic reward navigation task. We found that combination of SR-based system learning mainly from positive RPEs and IR-based system learning mainly from negative RPEs could achieve a good performance in the task, as compared to other combinations. In such a combination of appetitive SR-based and aversive IR-based systems, both systems show activities of comparable magnitudes with opposite signs, consistent with the suggested profiles of the two BG pathways. Moreover, the architecture of such a combination provides a novel coherent explanation for the functional significance and underlying mechanism of diverse findings about the cortico-BG circuits. These results suggest that particularly combining different representations with appetitive and aversive learning could be an effective learning strategy in certain dynamic environments, and it might actually be implemented in the cortico-BG circuits.Significance StatementAnimals can learn the value of states/actions from both positive and negative feedbacks. For learning, animals need to represent each state/action, individually (like representing a person by her/his identity only) or in a relation-based manner (like representing a person by friends or descendants). Different brain circuits may learn from positive and negative feedbacks with different rates, and may represent states/actions in different ways. We explored what combination of the feedback valence-dependent learning rates and the ways of state representation performs well in a dynamic reward navigation task. We found that a particular combination performed well, and we propose that several known anatomical and physiological properties of the cortico-basal ganglia circuits may indicate implementation of such a combination.

Opponent Learning with Different Representations in the Cortico-Basal Ganglia Pathways Can Develop Obsession-Compulsion Cycle

“…We examined how the dual-system agent having coupled SR- and IR-based systems (Figure 1C), originally developed in [35], behaved in this environmental model. As mentioned in the Introduction (and also in [35]), this dual-system agent was motivated by the suggestions that humans/animals use both model-based (SR-based) and model-free (IR-based) controls, which may learn differently from positive and negative RPEs through uneven projections from different cortical populations hosting different representations (SR or IR) to the direct and indirect pathways of basal ganglia. In this dual-system agent, each system develops the system-specific value of each action, and the average of the two system-specific values, named the (integrated) action value, is used for soft- max action selection and calculation of RPEs.…”

“…As described in the Introduction, the opponent combination of appetitive SR- and aversive IR-based systems has recently been shown to perform well in certain dynamic environments [35]. Specifically, in a virtual reward navigation task in which reward location dynamically changed, the appetitive SR + aversive IR agent outperformed the agents with other combinations.…”

“…The dual-system agent having a coupled successor-representation(SR)-based system and individual-representation(IR)-based system, adopted from the previous study [35]. (A) Schematic illustrations of SR (left) and IR (right).…”

“…(B) Different cortical regions/populations having SR or IR may unevenly target/activate the direct and indirect pathways of basal ganglia (BG), which have been suggested to be crucial for learning from positive and negative feedbacks, respectively. The line widths of the arrows indicate the suggested preferences of projections/activations described in the Introduction and [35]. (C) The dual-system agent incorporating the potentially uneven projections/activations from the cortical regions/populations having SR or IR to the BG pathways.…”

“…Given these, it is conceivable that SR-based system and IR-based system differentially learn from positive and negative RPEs in the brain (Figure 1C). Recent work [35] explored this possibility, and found that combination of SR- and IR-based systems learning mainly from positives and negative RPEs, respectively (referred to as the appetitive SR + aversive IR agent), performed well in certain dynamic reward environments. This work further suggested that implementation of such a combination in the cortico-basal ganglia pathways seems in line with various anatomical and physiological findings, including activations indicative of SR in limbic or visual cortices [17, 18] and preferential connections from limbic and visual cortices to the direct pathway of basal ganglia [20, 21].…”