In recent years, ideas from the computational field of reinforcement learning have revolutionized the study of learning in the brain, famously providing new, precise theories of how dopamine affects learning in the basal ganglia. However, reinforcement learning algorithms are notorious for not scaling well to multidimensional environments, as is required for real-world learning. We hypothesized that the brain naturally reduces the dimensionality of real-world problems to only those dimensions that are relevant to predicting reward, and conducted an experiment to assess by what algorithms and with what neural mechanisms this "representation learning" process is realized in humans. Our results suggest that a bilateral attentional control network comprising the intraparietal sulcus, precuneus, and dorsolateral prefrontal cortex is involved in selecting what dimensions are relevant to the task at hand, effectively updating the task representation through trial and error. In this way, cortical attention mechanisms interact with learning in the basal ganglia to solve the "curse of dimensionality" in reinforcement learning.
SUMMARY
Little is known about the relationship between attention and learning during decision making. Using eye tracking and multivariate pattern analysis of fMRI data, we measured participants’ dimensional attention as they performed a trial-and-error learning task in which only one of three stimulus dimensions was relevant for reward at any given time. Analysis of participants’ choices revealed that attention biased both value computation during choice and value update during learning. Value signals in the ventromedial prefrontal cortex and prediction errors in the striatum were similarly biased by attention. In turn, participants’ focus of attention was dynamically modulated by ongoing learning. Attentional switches across dimensions correlated with activity in a frontoparietal attention network, which showed enhanced connectivity with the ventromedial prefrontal cortex between switches. Our results suggest a bidirectional interaction between attention and learning: attention constrains learning to relevant dimensions of the environment, while we learn what to attend to via trial and error.
Reinforcement learning enables organisms to adjust their behavior in
order to maximize rewards. Electrophysiological recordings of dopaminergic
midbrain neurons have shown that they code the difference between actual and
predicted rewards, i.e., the reward prediction error, in many species. This
error signal is conveyed to both the striatum and cortical areas and is thought
to play a central role in learning to optimize behavior. However, in human daily
life rewards are diverse and often only indirect feedback is available. Here we
explore the range of rewards that are processed by the dopaminergic system in
human participants, and examine whether it is also involved in learning in the
absence of explicit rewards. While results from electrophysiological recordings
in humans are sparse, evidence linking dopaminergic activity to the metabolic
signal recorded from the midbrain and striatum with functional magnetic
resonance imaging (fMRI) is available. Results from fMRI studies suggest that
the human ventral striatum (VS) receives valuation information for a diverse set
of rewarding stimuli. These range from simple primary reinforcers such as juice
rewards over abstract social rewards to internally generated signals on
perceived correctness, suggesting that the VS is involved in learning from
trial-and-error irrespective of the specific nature of provided rewards. In
addition, we summarize evidence that the VS can also be implicated when learning
from observing others, and in tasks that go beyond simple
stimulus-action-outcome learning, indicating that the reward system is also
recruited in more complex learning tasks.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.