Neural signatures of reinforcement learning correlate with strategy adoption during spatial navigation

Abstract

Human navigation is generally believed to rely on two types of strategy adoption, route-based and map-based strategies. Both types of navigation require making spatial decisions along the traversed way although formal computational and neural links between navigational strategies and mechanisms of value-based decision making have so far been underexplored in humans. Here we employed functional magnetic resonance imaging (fMRI) while subjects located different objects in a virtual environment. We then modelled their paths using reinforcement learning (RL) algorithms, which successfully explained decision behavior and its neural correlates. Our results show that subjects used a mixture of route and map-based navigation and their paths could be well explained by the model-free and model-based RL algorithms. Furthermore, the value signals of model-free choices during route-based navigation modulated the BOLD signals in the ventro-medial prefrontal cortex (vmPFC), whereas the BOLD signals in parahippocampal and hippocampal regions pertained to model-based value signals during map-based navigation. Our findings suggest that the brain might share computational mechanisms and neural substrates for navigation and value-based decisions such that model-free choice guides route-based navigation and model-based choice directs map-based navigation. These findings open new avenues for computational modelling of wayfinding by directing attention to value-based decision, differing from common direction and distances approaches.

Striatal and hippocampal contributions to flexible navigation in rats and humans

“…In a more recent virtual navigation task, Anggraini et al (2018) identified model-free correlates in dorsal striatum. Model-based correlates were found in the parahippocampus and overlapped with model-free correlates in the retrosplenial cortex.…”

Reinforcement learning approaches to hippocampus-dependent flexible spatial navigation

“…However, as they contain exhaustive information about the available routes between states, they are more flexible towards changing goal locations than model-free approaches ( Keramati et al, 2011 ). A study of spatial navigation in human participants showed that, although paths to the goal were shorter, choice times were higher in trials when the behaviour matches that of a model-based agent compared to trials where it matches that of a model-free agent ( Anggraini et al, 2018 ). In studies involving rats in a T-maze, vicarious trial and error (VTE) behaviour, short pauses that rats make at decision points, tend to get shorter with repetitive exposure to the same goal location ( Redish, 2016 ).…”

“…First, unlike what would be expected based on model-based mechanisms, rats do not reach optimal trajectories in the DMP task, as reflected by the observation that escape latencies on trials 2 to 4 remain higher than would typically be observed following incremental place learning in the watermaze (compare Morris et al (1986) , Steele and Morris (1999) and Bast et al (2009) ) and than would be expected from a model-based agent ( Sutton and Barto, 2018 ). Findings in humans by Anggraini et al (2018) suggest that participants that used more model-based approaches more often took the shortest path to goal locations. Second, classical model-based approaches are currently mostly implemented in discrete state space, although they can be approximated to continuous spaces ( Jong and Stone, 2007 ).…”

The Neuroscience of Spatial Navigation and the Relationship to Artificial Intelligence

“…In contrast, when the hippocampus is involved, faster one-shot associative learning rules are applied to solve spatial navigation. Recent studies in humans link these mechanisms for decision making, in which model-free choice guides route-based navigation and model-based choice directs map-based navigation (Anggraini et al, 2018 ).…”