Cortical mechanisms for reinforcement learning in competitive games

Seo, Hyojung; Lee, Dongsoo

doi:10.1098/rstb.2008.0158

Cited by 61 publications

(65 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thorndike's "law of effect" is no longer in operation, even though the "spread of effect" of reward (19) still operates. Although several representations of choice history and reward history exist in the brain (20), only a limited number of regions, including the lOFC, represent the conjoint history of choices and rewards (21,22). An lOFC lesioned animal should struggle to learn when each option's value is very different from the others because it will erroneously estimate the value of each option as close to the mean.…”

Section: Discussionmentioning

confidence: 99%

Separate value comparison and learning mechanisms in macaque medial and lateral orbitofrontal cortex

Noonan

Walton

Behrens

et al. 2010

Proc. Natl. Acad. Sci. U.S.A.

332

386

View full text Add to dashboard Cite

Uncertainty about the function of orbitofrontal cortex (OFC) in guiding decision-making may be a result of its medial (mOFC) and lateral (lOFC) divisions having distinct functions. Here we test the hypothesis that the mOFC is more concerned with reward-guided decision making, in contrast with the lOFC's role in reward-guided learning. Macaques performed three-armed bandit tasks and the effects of selective mOFC lesions were contrasted against lOFC lesions. First, we present analyses that make it possible to measure reward-credit assignment-a crucial component of reward-value learning-independently of the decisions animals make. The mOFC lesions do not lead to impairments in reward-credit assignment that are seen after lOFC lesions. Second, we examined how the reward values of choice options were compared. We present three analyses, one of which examines reward-guided decision making independently of reward-value learning. Lesions of the mOFC, but not the lOFC, disrupted reward-guided decision making. Impairments after mOFC lesions were a function of the multiple option contexts in which decisions were made. Contrary to axiomatic assumptions of decision theory, the mOFC-lesioned animals' value comparisons were no longer independent of irrelevant alternatives.A lthough it is widely agreed that the orbitofrontal cortex (OFC) is important for value-guided behavior (1, 2), the precise nature of its contribution is unclear. On the one hand, neuroimaging experiments suggest its ventromedial aspect is part of the circuit for deciding between options on the basis of their value (3, 4). Moreover De Martino et al. (5) have reported that ventromedial frontal activity is correlated with the degree to which subjects' decisions are rational and uninfluenced by irrelevant features of the context in which they are made. In contrast, other accounts emphasize a role in learning and storage of option values (6).One potential reason for lack of consensus about the OFC may be that there is functional specialization within it, as suggested by the distinct anatomical connections of its ventromedial and lateral divisions (7) The aim of our study was, therefore, to assess evidence of functional specialization within the OFC for value learning and value-guided decision making. The effect of medial OFC (mOFC) lesions was compared with that of lateral OFC (lOFC) lesions in two groups of animals.The investigation of value-guided decision making was based on the premise that a lesion that disrupts part a circuit for deciding and discriminating between choice options of differing value should impair decision making as a function of the proximity of options' values. By analogy, lesions of visual feature-discrimination mechanisms disrupt discrimination as a function of the similarity of the visual features (8). In experiments reported below, macaques were trained to choose between three stimuli associated with differing probabilities of reward. The probabilities of reward, therefore, determined the options' values and the proximity of the best (...

show abstract

Section: Discussionmentioning

confidence: 99%

Separate value comparison and learning mechanisms in macaque medial and lateral orbitofrontal cortex

Noonan

Walton

Behrens

et al. 2010

Proc. Natl. Acad. Sci. U.S.A.

332

386

View full text Add to dashboard Cite

show abstract

“…As another possibility, which is not mutually exclusive to the scenario above, the estimate of reward-based arming probability (i.e., action value function estimated according to a simple RL algorithm) and the latest run length might be separately computed before being combined to estimate the final stacked arming probability. Physiological studies have found neural signals that are related to action value functions that were computed based on a simple RL algorithm Samejima et al 2005;Seo and Lee 2007) and neural signals that are related to animal's previous choice Kim et al 2007;Seo and Lee 2008) or the number of selfexecuted actions (Sawamura et al 2002) in cortical and subcortical brain structures. The latter may represent the latest run length in the DAWH task.…”

Section: Discussionmentioning

confidence: 99%

Model-based reinforcement learning under concurrent schedules of reinforcement in rodents

Huh

Jo²,

Kim³

et al. 2009

Learn. Mem.

View full text Add to dashboard Cite

Reinforcement learning theories postulate that actions are chosen to maximize a long-term sum of positive outcomes based on value functions, which are subjective estimates of future rewards. In simple reinforcement learning algorithms, value functions are updated only by trial-and-error, whereas they are updated according to the decision-maker's knowledge or model of the environment in model-based reinforcement learning algorithms. To investigate how animals update value functions, we trained rats under two different free-choice tasks. The reward probability of the unchosen target remained unchanged in one task, whereas it increased over time since the target was last chosen in the other task. The results show that goal choice probability increased as a function of the number of consecutive alternative choices in the latter, but not the former task, indicating that the animals were aware of time-dependent increases in arming probability and used this information in choosing goals. In addition, the choice behavior in the latter task was better accounted for by a model-based reinforcement learning algorithm. Our results show that rats adopt a decision-making process that cannot be accounted for by simple reinforcement learning models even in a relatively simple binary choice task, suggesting that rats can readily improve their decision-making strategy through the knowledge of their environments.Animals must continually update their behavioral strategies according to changes in an environment in order to optimize their choices. Reinforcement learning (RL) models (Sutton and Barto 1998) provide a powerful theoretical framework for understanding choice behavior in humans and animals in a dynamic environment. In theories of RL, future actions are chosen so as to maximize a long-term sum of positive outcomes, and this can be accomplished by a set of value functions that represent the amount of expected reward that is associated with particular states or actions. The value functions are continually updated based on the reward prediction error, which is the difference between the expected and actual rewards. This way, even without prior knowledge about an uncertain and dynamically changing environment, an animal can discover the structure of the environment that can be exploited for optimal choice by trial-and-error. Not surprisingly, human and monkey choice behaviors in various tasks are well described by reinforcement learning algorithms (e.g., O'Doherty et al. 2003;Barraclough et al. 2004;Lee et al. 2004;Samejima et al. 2005;Daw et al. 2006;Pessiglione et al. 2006).The updating of value functions can be achieved in two fundamentally different ways. In simple or direct RL algorithms, value functions are updated only by trial-and-error. In other words, only the value function that is associated with the chosen action is updated, and those that are associated with uncommitted actions remain unchanged. On the other hand, in indirect or model-based RL algorithms, the value functions might also change according to the decis...

show abstract

“…Neural activity (dependent variable) was also correlated across successive trials (e.g., mean serial correlation ϭ 0.052 Ϯ 0.007 and 0.062 Ϯ 0.007 for the neural data during the first and last 1 s of the delay stage), which could be due to a number of factors, such as a slow drift in the spike rates during the recording session. Regardless of its origin, such serial correlation in spike rates could potentially violate the independence assumption in the regression analysis and increase the amount of activity spuriously correlated with action values (Seo and Lee, 2008). We therefore used a permutation test to evaluate statistical significance of regression coefficients for the multiple regression analyses that contained action values.…”

Section: (Model 10)mentioning

confidence: 99%

Role of Striatum in Updating Values of Chosen Actions

Kim

Sul²,

Huh³

et al. 2009

J. Neurosci.

195

304

View full text Add to dashboard Cite

The striatum is thought to play a crucial role in value-based decision making. Although a large body of evidence suggests its involvement in action selection as well as action evaluation, underlying neural processes for these functions of the striatum are largely unknown. To obtain insights on this matter, we simultaneously recorded neuronal activity in the dorsal and ventral striatum of rats performing a dynamic two-armed bandit task, and examined temporal profiles of neural signals related to animal's choice, its outcome, and action value. Whereas significant neural signals for action value were found in both structures before animal's choice of action, signals related to the upcoming choice were relatively weak and began to emerge only in the dorsal striatum ϳ200 ms before the behavioral manifestation of the animal's choice. In contrast, once the animal revealed its choice, signals related to choice and its value increased steeply and persisted until the outcome of animal's choice was revealed, so that some neurons in both structures concurrently conveyed signals related to animal's choice, its outcome, and the value of chosen action. Thus, all the components necessary for updating values of chosen actions were available in the striatum. These results suggest that the striatum not only represents values associated with potential choices before animal's choice of action, but might also update the value of chosen action once its outcome is revealed. In contrast, action selection might take place elsewhere or in the dorsal striatum only immediately before its behavioral manifestation.

show abstract

Cortical mechanisms for reinforcement learning in competitive games

Cited by 61 publications

References 63 publications

Separate value comparison and learning mechanisms in macaque medial and lateral orbitofrontal cortex

Separate value comparison and learning mechanisms in macaque medial and lateral orbitofrontal cortex

Model-based reinforcement learning under concurrent schedules of reinforcement in rodents

Role of Striatum in Updating Values of Chosen Actions

Contact Info

Product

Resources

About