Model-based spatial navigation in the hippocampus-ventral striatum circuit: A computational analysis

Abstract

While the neurobiology of simple and habitual choices is relatively well known, our current understanding of goal-directed choices and planning in the brain is still limited. Theoretical work suggests that goal-directed computations can be productively associated to model-based (reinforcement learning) computations, yet a detailed mapping between computational processes and neuronal circuits remains to be fully established. Here we report a computational analysis that aligns Bayesian nonparametrics and model-based reinforcement learning (MB-RL) to the functioning of the hippocampus (HC) and the ventral striatum (vStr)–a neuronal circuit that increasingly recognized to be an appropriate model system to understand goal-directed (spatial) decisions and planning mechanisms in the brain. We test the MB-RL agent in a contextual conditioning task that depends on intact hippocampus and ventral striatal (shell) function and show that it solves the task while showing key behavioral and neuronal signatures of the HC—vStr circuit. Our simulations also explore the benefits of biological forms of look-ahead prediction (forward sweeps) during both learning and control. This article thus contributes to fill the gap between our current understanding of computational algorithms and biological realizations of (model-based) reinforcement learning.

Making the Environment an Informative Place: A Conceptual Analysis of Epistemic Policies and Sensorimotor Coordination

“…This renders cognitive agents active entities that constantly seek evidence for their plans and their goals, rather than passive entities that try to mirror the external environment via their internal models [ 37 , 38 , 39 , 40 , 41 ]. Note that the model-based agent exemplifies the usage of an “explicit” model—it has an explicit, content-involving representation of its task, organized as an internal model (e.g., a map of spatial locations, reminiscent of hippocampal spatial codes [ 42 , 43 , 44 , 45 ]). Furthermore, it uses the model for explicit predictive inference (i.e., to predict future locations and associated rewards) that permits “scoring” the quality of candidate policies and ultimately selecting amongst them.…”

Multi-step planning in the brain

“…Internal decisions must also be made regarding which sequences of states should be considered. Theoretical accounts propose that the hippocampus does not perform all of these functions on its own, but rather that hippocampus interacts with other brain regions, including prefrontal cortex, ventral striatum, and OFC [108][109][110] . Recent data have added to the evidence for this view, with one study showing specific behavioral correlates for replay events that were coordinated between hippocampus and prefrontal cortex [111] , and another study showing that silencing prefrontal cortex caused disruptions in theta sequences [112] .…”

The Neuroscience of Spatial Navigation and the Relationship to Artificial Intelligence

“…In addition, recent work has shown that retrosplenial neurons exhibit HD, position, and spike in the relation to the animals distance between path segments, as well as a conjunctive combination of these firing characteristics (Alexander and Nitz, 2015 ; Mao et al, 2017 , 2018 ). Thus, the parietal and retrosplenial cortex may be part of a circuit that interfaces between allocentric and egocentric frames of reference (Pennartz et al, 2011 ; Stoianov et al, 2018 ). Therefore, these computations performed in the parietal and retrosplenial cortex might be crucial for understanding how transformations between self-centered experiences is related to map-like representations of space.…”

“…In AI and neuroscience, research in RL and rodent spatial navigation has already proved to be a successful approach to understand how these two concepts are closely related and the interaction between these fields can help to inform one another. For example (Stoianov et al, 2018 ), demonstrated how a RL model can replicate results in rodent experiments in which contextual cues are manipulated to explore the behavioral and brain constrains in goal directed navigation tasks.…”