The Role of the Ventromedial Prefrontal Cortex in Abstract State-Based Inference during Decision Making in Humans

212

229

The ability to rapidly and flexibly adapt decisions to available rewards is crucial for survival in dynamic environments. Rewardbased decisions are guided by reward expectations that are updated based on prediction errors, and processing of these errors involves dopaminergic neuromodulation in the striatum. To test the hypothesis that the COMT gene Val 158 Met polymorphism leads to interindividual differences in reward-based learning, we used the neuromodulatory role of dopamine in signaling prediction errors. We show a behavioral advantage for the phylogenetically ancestral Val/Val genotype in an instrumental reversal learning task that requires rapid and flexible adaptation of decisions to changing reward contingencies in a dynamic environment. Implementing a reinforcement learning model with a dynamic learning rate to estimate prediction error and learning rate for each trial, we discovered that a higher and more flexible learning rate underlies the advantage of the Val/Val genotype. Model-based fMRI analysis revealed that greater and more differentiated striatal fMRI responses to prediction errors reflect this advantage on the neurobiological level. Learning rate-dependent changes in effective connectivity between the striatum and prefrontal cortex were greater in the Val/Val than Met/Met genotype, suggesting that the advantage results from a downstream effect of the prefrontal cortex that is presumably mediated by differences in dopamine metabolism. These results show a critical role of dopamine in processing the weight a particular prediction error has on the expectation updating for the next decision, thereby providing important insights into neurobiological mechanisms underlying the ability to rapidly and flexibly adapt decisions to changing reward contingencies.COMT ͉ dopamine ͉ functional MRI ͉ learning rate ͉ reinforcement learning H uman learning spans a wide range of functions from ancient evolutionary accomplishments [e.g., fast and intuitive learning from rewards (1, 2)], to complex executive processes that require high capacities of attention and working memory (3, 4). These functions are supported by interacting brain regions that comprise phylogenetically ancient structures (e.g., the striatum) as well as more recently evolved neocortical structures [e.g., the prefrontal cortex (PFC)]. Still, different learning processes rely on the same neuromodulators. Dopamine (DA) modulates synaptic efficacy in both the striatum and PFC (5) and is thus involved in different kinds of learning. Accordingly, interindividual differences in dopaminergic neuromodulatory mechanisms contribute to performance differences in both simple instrumental learning tasks and complex cognitive tasks (6-10).One interindividual difference in dopaminergic neuromodulation that has been linked to PFC function is the uniquely human Val 158 Met polymorphism in the enzyme catechol-O-methyltransferase (COMT), which regulates extrasynaptical DA degradation (8-10). COMT activity is lower in individuals homozygous for the mutated allele...

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Genetic variation in dopaminergic neuromodulation influences the ability to rapidly and flexibly adapt decisions

Krugel

Biele

Mohr

et al. 2009

212

229

“…A major function of the chosen value signals in these areas could be to facilitate the generation of a prediction error signal that can then be used to update future action values. It is notable that the two signals required to compute a prediction error, namely the actual outcome and the expected outcome (of the chosen action) are both represented in ventromedial prefrontal cortex (8,9,41,42). Therefore this region is ideally placed to facilitate computation of the prediction error signal that could then be transferred on to dopaminergic neurons in the midbrain for subsequent broadcast (43,44).…”

Section: Discussionmentioning

confidence: 99%

“…This is central to understand how the brain makes action-based choices. For example, a number of chosen value signals have been found in the orbital and medial prefrontal cortex (8,9) and amygdala (10,11). Note that these signals are quite distinct from action values, and are not precursors to choice, because they reflect the value of the actions that were selected in the decision.…”

mentioning

confidence: 99%

Neural computations underlying action-based decision making in the human brain

Wunderlich

Rangel

O’Doherty

2009

271

214

Action-based decision making involves choices between different physical actions to obtain rewards. To make such decisions the brain needs to assign a value to each action and then compare them to make a choice. Using fMRI in human subjects, we found evidence for action-value signals in supplementary motor cortex. Separate brain regions, most prominently ventromedial prefrontal cortex, were involved in encoding the expected value of the action that was ultimately taken. These findings differentiate two main forms of value signals in the human brain: those relating to the value of each available action, likely reflecting signals that are a precursor of choice, and those corresponding to the expected value of the action that is subsequently chosen, and therefore reflecting the consequence of the decision process. Furthermore, we also found signals in the dorsomedial frontal cortex that resemble the output of a decision comparator, which implicates this region in the computation of the decision itself.acc ͉ action value ͉ reinforcement learning ͉ sma ͉ vmpfc C onsider a goalkeeper trying to stop a soccer ball during a penalty kick. Within a brief amount of time he needs to choose between jumping to the left or right goal posts. Repeated play against the same opponents allows him to learn about their scoring tendencies, which can be used to compute the values of a left and a right jump before making a decision. It is a long-established view in economics, psychology, and computational neuroscience that the brain makes choices among actions by first computing a value for each possible action, and then selecting one of them on the basis of those values (1-3). This raises two fundamental questions in decision neuroscience: (1) where in the brain are the values of different types of actions encoded? and (2) how and where does the brain compare those values to generate a choice?An emerging theme in decision neuroscience is that organisms need to make a number of value-related computations to make even simple choices (4). Consider the case of action-based choice exemplified by the goalkeeper's problem. First, he needs to assign a value to each action under consideration. These signals, known as action values, encode the value of each action before choice and regardless of whether it is subsequently chosen or not, which allows them to serve as inputs into the decision-making process (5-7). Second, these action values are compared to generate a choice.

“…The lack of empirical affirmation that such regions exist has made it difficult to provide a mechanistic account of how different sources of probabilistic evidence are combined to compute value. Despite the fact that several studies on valuebased decision making have consistently implicated the medial prefrontal cortex in encoding expected value signals (18)(19)(20)(21)(22)(23)(24)(25)(26)(27), it remains unknown whether it is directly involved in computing the value signal needed to make the decision (by combining different sources of probabilistic evidence) or whether it merely reflects the consequence of the decision process.…”

mentioning

confidence: 99%

A mechanistic account of value computation in the human brain

Philiastides

Biele

Heekeren

2010

129

104

To make decisions based on the value of different options, we often have to combine different sources of probabilistic evidence. For example, when shopping for strawberries on a fruit stand, one uses their color and size to infer-with some uncertainty-which strawberries taste best. Despite much progress in understanding the neural underpinnings of value-based decision making in humans, it remains unclear how the brain represents different sources of probabilistic evidence and how they are used to compute value signals needed to drive the decision. Here, we use a visual probabilistic categorization task to show that regions in ventral temporal cortex encode probabilistic evidence for different decision alternatives, while ventromedial prefrontal cortex integrates information from these regions into a value signal using a difference-based comparator operation.I n our everyday lives we often have to combine different sources of probabilistic information to make decisions that are more likely to produce desirable outcomes. For instance, imagine choosing strawberries on a fruit stand on the basis of their size and color, trying to increase the likelihood your strawberry smoothie will taste delicious. Despite much progress in understanding the neural systems that mediate reward-and value-based decision making in humans (1-8), including recent reports of value-based modulations in sensory cortex (9-11), it remains unclear how the brain represents different sources of probabilistic information and how they are used to compute the value signal necessary to make a decision.Research on perceptual decision making has already established that category-selective regions in sensory cortex encode the amount of perceptual information (i.e., sensory evidence) used in the decision process (12-17). To date it remains unclear, however, whether sensory regions also represent the amount of probabilistic reward information (i.e., probabilistic evidence) associated with different decision alternatives during value-based decisions. The lack of empirical affirmation that such regions exist has made it difficult to provide a mechanistic account of how different sources of probabilistic evidence are combined to compute value. Despite the fact that several studies on valuebased decision making have consistently implicated the medial prefrontal cortex in encoding expected value signals (18-27), it remains unknown whether it is directly involved in computing the value signal needed to make the decision (by combining different sources of probabilistic evidence) or whether it merely reflects the consequence of the decision process.Notably, the only available empirical evidence that could provide mechanistic insights into the computation of choice values comes from work on perceptual decision making. Specifically, this line of research has shown that, for binary perceptual choices, decision variables are computed by integrating the difference of the outputs of neural populations, tuned to sensory evidence for each decision alternative (12)(13)(14...