Neurocomputational mechanisms of reinforcement-guided learning in humans: A review

Abstract

Our world is filled with uncertainty and limited resources. Nearly every decision we face is clouded in uncertainty-uncertainty about the consequences of our decisions, uncertainty about whether others will obtain resources before we do, uncertainty about how different individuals will respond in similar contexts. Fortunately, we are often faced with the same or similar decision problems repeatedly, providing the opportunity to learn from our previous decisions and outcomes and to dynamically adapt decision-making strategies. The ability to use reinforcements flexibly to modify goal-directed behavior in the service of maximizing rewards and minimizing punishments is termed reinforcement learning. Theories and computational models of reinforcement learning provide a biologically and mathematically grounded framework within which to characterize, predict, and understand the b behavioral, cognitive, and neural bases of reward-guided decision making. The brain is highly adept at reinforcement learning, and although the neural processes that support this adaptation are being discovered, much remains unknown about the mechanisms by which rewards and punishments are used by the brain to optimize and guide decision making. Recent research in neuroscience and computational modeling suggests that theories of reinforcement learning provide a useful framework within which to study the behavioral and neural mechanisms of reward-based learning and decision making Camerer, 2003;Dayan & Balleine, 2002;Egelman, Person, & Montague, 1998;Montague & Berns, 2002;O'Doherty, Hampton, & Kim, 2007;Schultz, Dayan, & Montague, 1997). This literature seems to be in its infancy and yet is making noteworthy strides in chart acterizing the brain systems and neural computations that support reinforcement learning.Imaging and single-unit recording studies have demonstrated that reinforcements elicit increases in neural activity in several brain regions-notably, those in midbrain-striatal-frontal cortical circuits. Dopamine appears to be critically involved in such processes, although its precise function is still debated (Berridge, 2007;Montague, Hyman, & Cohen, 2004;Redgrave & Gurney, 2006;Schultz, 1998). Generally, changes in brain activity strongly correlate with reinforcements and their predictors, with some regions exhibiting increases in activity following rewards, as compared with losses, and some regions exhibiting increases in activity following losses, as compared with rewards. One might ask the simple question, why? Why should these brain circuits "care about" reinforcements? If the reinforcement already occurred, why continue processing and evaluatr ing its relative reward value after it has been consumed or received? The straightforward and intuitive answer is that reinforcements are useful for guiding future decisions and t actions. That is, reinforcements provide information that can be used to adjust behavior to maximize future rewards f when similar conditions or decision problems arise. Most of the neuroscience studies of rein...

Life Span Differences in Electrophysiological Correlates of Monitoring Gains and Losses during Probabilistic Reinforcement Learning

“…Over the past decade, a great deal of progress has been made toward understanding the neural substrates of reward detection and reward expectation and the adjustment of future decisions on the basis of reward history (for reviews, see Cohen, 2008;Schultz, 2007). Relevant findings have been reported at neuroanatomical, neurochemical, and electrophysiological levels.…”

“…Relevant findings have been reported at neuroanatomical, neurochemical, and electrophysiological levels. The functional contributions of the BG, amygdala, OFC, and ACC to decisionmaking, reward processing, and reinforcement learning are Max Planck Institute for Human Development, Berlin, Germany being increasingly understood (Cohen, 2008;Redgrave, Gurney, & Reynolds, 2008;Kennerley, Walton, Behrens, Buckley, & Rushworth, 2006;Ridderinkhof, Ullsperger, Crone, & Nieuwenhuis, 2004;Walton, Devlin, & Rushworth, 2004;Walton, Bannerman, & Rushworth, 2002;Schultz, 2000;Shima & Tanji, 1998). Specifically, ACC is conceived of as a control structure or as a structure that integrates positive and negative aspects of an action to orient future behavior (cf., Rushworth & Behrens, 2008;.…”

High-Learners Present Larger Mid-Frontal Theta Power and Connectivity in Response to Incorrect Performance Feedback

“…Beyond the FRN, neuronal oscillations can provide complementary information on the various features of feedback-guided learning Christie and Tata, 2009;van de Vijver et al, 2011). Theta power (4 -8 Hz) increases from 200 to 500 ms following negative feedback Cavanagh et al, 2010Cavanagh et al, , 2012a, while beta power (15-30 Hz) increases from 200 to 500 ms following positive feedback Marco-Pallares et al, 2008;van de Vijver et al, 2011).…”

Neural Correlates of Reinforcement Learning and Social Preferences in Competitive Bidding

“…3, Table 2). These areas have been associated with reward prediction errors in reinforcement learning numerously (for review, see Cohen, 2008).…”