Subcortical Substrates of Explore-Exploit Decisions in Primates

Costa, Vincent D.; Mitz, Andrew R.; Averbeck, Bruno B.

doi:10.1016/j.neuron.2019.05.017

Cited by 117 publications

(140 citation statements)

References 85 publications

Supporting

Mentioning

126

Contrasting

Order By: Relevance

“…Whereas previous studies of the explore/exploit dilemma have primarily focused on the neocortex, striatum and amygdala (Badre et al, 2012;Beharelle et al, 2015;Blanchard and Gershman, 2018;Costa et al, 2019;Daw et al, 2006), we show that the hippocampus plays a key role in resolving this dilemma when values are organized spatially. Capturing this organization with a basis function RL model of exploration/exploitation in a unidimensional continuous space, we observed doubly dissociated representations of reinforcement along the hippocampal long axis: state-wise RPE signals in the PH facilitating exploration and global value maximum signals in the AH driving the transition to exploitation.…”

Section: Discussioncontrasting

confidence: 67%

“…Many studies of exploration have used reinforcement learning (RL) models to examine decision-making on discrete choice multi-armed bandit tasks (Gershman, 2018;Payzan-LeNestour and Bossaerts, 2011). These experiments have provided insights into the neural correlates of exploration, including the prefrontal and cingulate cortex (Badre et al, 2012;Beharelle et al, 2015;Blanchard and Gershman, 2018;Daw et al, 2006), as well as the striatum and amygdala (Costa et al, 2019). Unlike simple discrete choice tasks, most real world environments have a complex spatial, temporal, or abstract structure (Liu et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Differential reinforcement encoding along the hippocampal long axis helps resolve the explore/exploit dilemma

Dombrovski

Luna

Hallquist

2020

Preprint

View full text Add to dashboard Cite

Hippocampal maps incorporate reward information, yet the functional contributions of the anterior and posterior hippocampal divisions (AH and PH) to reinforcement learning remain unclear. Here, we examined exploration and exploitation of a continuous unidimensional task with a basis function reinforcement learning model. In model-based fMRI analyses, we found doubly dissociated representations along the hippocampal long axis: state-wise reward prediction error signals in the PH (tail) and global value maximum signals in the AH (anterior body). PH prediction error signals predicted exploration whereas AH global value maximum signals predicted exploitation. AH-mediated exploitation depended on value representations compressed across episodes and options. PH responses to reinforcement were early and phasic while AH responses were delayed and evolved throughout learning. During choice, AH (head) displayed goal cell-like responses to the global value maximum. In summary, granular reinforcement representations in PH facilitate exploration and compressed representations of the value maximum in AH facilitate exploitation.

show abstract

Section: Discussioncontrasting

confidence: 67%

Section: Introductionmentioning

confidence: 99%

Differential reinforcement encoding along the hippocampal long axis helps resolve the explore/exploit dilemma

Dombrovski

Luna

Hallquist

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…These studies have shown that the amygdala plays an important role in the associative process between cues and shocks. Although, the amygdala is also involved in reward guided behavior (Costa, Dal Monte et al 2016, Averbeck and Costa 2017, Costa, Mitz et al 2019. We did find a small, although significant, population of amygdala neurons, that encoded the auditory cue and the auditory target.…”

Section: Discussionmentioning

confidence: 60%

“…The present study also shows a substantial dissociation of function between the BLA and dlPFC. This dissociation differs from the similarity between these structures seen in reinforcement learning (RL) tasks, in which both dlPFC and the BLA show substantial encoding of the identity of visual stimuli, the reward values associated with those stimuli, and reward outcomes (Bartolo, Saunders et al 2019, Costa, Mitz et al 2019). The primary difference between the BLA and dlPFC, in RL tasks, is that the dlPFC strongly encodes the direction of eye movements required to saccade to a rewarding visual stimulus (Bartolo, Saunders et al 2019), whereas the BLA encodes eye movement directions only at a low level (Costa, Mitz et al 2019).…”

Section: Discussionmentioning

confidence: 76%

“…This dissociation differs from the similarity between these structures seen in reinforcement learning (RL) tasks, in which both dlPFC and the BLA show substantial encoding of the identity of visual stimuli, the reward values associated with those stimuli, and reward outcomes (Bartolo, Saunders et al 2019, Costa, Mitz et al 2019). The primary difference between the BLA and dlPFC, in RL tasks, is that the dlPFC strongly encodes the direction of eye movements required to saccade to a rewarding visual stimulus (Bartolo, Saunders et al 2019), whereas the BLA encodes eye movement directions only at a low level (Costa, Mitz et al 2019). Thus, in RL tasks, the BLA and dlPFC show similar responses, which are also similar to those seen in the ventral striatum (Costa, Mitz et al 2019) and orbito-frontal cortex (Costa and Averbeck 2020), with which the BLA is mono-synaptically connected.…”

Section: Discussionmentioning

confidence: 76%

See 1 more Smart Citation

Correlates of auditory decision making in prefrontal, auditory, and basal lateral amygdala cortical areas

Napoli

Camalier

Brown

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Auditory selective listening and decision making underlies important processes, including attending to a single speaker in a crowded room, often referred to as the cocktail party problem. To examine the neural mechanisms underlying these behaviors, we developed a novel auditory selective listening paradigm for monkeys. In this task, monkeys had to detect a difficult to discriminate target embedded in noise when presented in a pre-cued location (either left or right) and ignore it if it was in the opposite location. While the animals carried out the task we recorded neural activity in primary auditory cortex (AC), dorsal lateral prefrontal cortex (dlPFC) and the basal lateral amygdala (BLA), given that these areas have been implicated in auditory decision making, selective listing, and/or rewardguided decision making. There were two main findings in the neural data. First, primary AC encoded the side of the cue and target, and the monkey's choice, before either dlPFC or the amygdala. The BLA encoded cue and target variables negligibly, but was engaged at the time of the monkey's choice. Second, decoding analyses suggested that errors followed primarily from a failure to encode the target stimulus in both AC and PFC, but earlier in AC. Thus, AC neural activity is poised to represent the sensory volley and decision making during selective listening before dlPFC, and they both precede activity in BLA.

show abstract

Analyzing Human Search Behavior When Subjective Returns are Unobservable

2023

View full text Add to dashboard Cite

The exploration versus exploitation dilemma is a critical issue in human information acquisition and sequential belief formation, and the multi-armed bandit problem has been widely used to address it. Because of its high descriptive accuracy, the SGU model, which combines SoftMax type probabilistic selection, Gaussian process regression type belief updating, and upper confidence interval type evaluation, has attracted much attention. However, this model assumes that the analyst has access to the returns from people’s choices, but in many realistic tasks, this assumption cannot be made because only choices are observable. Moreover, many of the returns are subjective. The authors introduce a new model-fitting method that overcomes this barrier and evaluates its performance using data sets derived from agent-based simulations and real consumer data. This approach has the potential to significantly broaden the range of issues to which the SGU model can be applied.

show abstract

Subcortical Substrates of Explore-Exploit Decisions in Primates

Cited by 117 publications

References 85 publications

Differential reinforcement encoding along the hippocampal long axis helps resolve the explore/exploit dilemma

Differential reinforcement encoding along the hippocampal long axis helps resolve the explore/exploit dilemma

Correlates of auditory decision making in prefrontal, auditory, and basal lateral amygdala cortical areas

Analyzing Human Search Behavior When Subjective Returns are Unobservable

Contact Info

Product

Resources

About