Goal-directed instrumental action: contingency and incentive learning and their cortical substrates

Balleine, Bernard W.; Dickinson, Anthony

doi:10.1016/s0028-3908(98)00033-1

Cited by 1,367 publications

(1,327 citation statements)

References 41 publications

Supporting

Mentioning

1,256

Contrasting

Unclassified

Order By: Relevance

“…Lesions or inactivations of all three areas have been found to cause impairments in devaluation and in a task in which knowledge about the contingencies associated with two lever-reward contingencies is tested (Balleine & Dickinson, 1998;Killcross & Coutureau, 2003;Yin, Ostlund, Knowlton, & Balleine, 2005), as well as in strategy set shifting (Block et al, 2007;Ragozzino, Detri ck, & Kesner, 1999;Ragozzino, Ragozzino, Mizumori, & Kesner, 2002). This suggests that the role of this loop may be in suppressing old associations so that new associations can be learned in tasks with new contingencies or responses, and that this can account for previous findings that have suggested a role for this circuit in "goal-directed action".…”

Section: Discussionmentioning

confidence: 99%

A limited role for mediodorsal thalamus in devaluation tasks.

Pickens

2008

Behavioral Neuroscience

View full text Add to dashboard Cite

Six experiments were performed to determine the role of mediodorsal thalamus (MD) in the devaluation task, varying the type of contingencies (Pavlovian or operant), the number of reinforcers (one versus two) and the order of experiments (in naïve or experimentally experienced rats). MD lesioned rats were impaired in devaluation performance when switched between Pavlovian and operant devaluation tasks, but not when switched from one Pavlovian devaluation task to another Pavlovian devaluation task. MD lesions caused no devaluation impairment in a multiple reinforcer Pavlovian devaluation task. These results suggest that MD lesions impair performance in devaluation tasks as a result of an inability to switch the form of associations made from one type of outcome-encoding association to another. This is in accord with previous literature suggesting that MD is needed for strategy set shifting. The results further suggest that MD is a necessary part of devaluation circuits only in cases in which previous associations need to be suppressed in order for new associations to be learned and control behavior, and otherwise the devaluation circuit does not require MD. KeywordsMediodorsal thalamus; devaluation; set-shifting; reward Devaluation tasks involve training a relationship that earns or predicts a valuable outcome, such as a lever press that earns a type of food or a light that predicts the delivery of food (Adams & Dickinson, 1981;Holland & Rescorla, 1975). The value of this outcome is then reduced by motivational (e.g., satiation) or associative methods (e.g., a food aversion is established by food-toxin pairings). After such outcome devaluation, responding based on the relationship that predicts or earns food is typically reduced. Similarities in task demands to human decision making, as well as similarities in neural underpinnings, suggests that the devaluation task provides a model of goal expectancy-guided behavior (as reviewed in Pickens & Holland, 2004).Many reports have examined the neural circuitry involved in acting according to the current value of goals. For example, lesions of amygdala, or specific lesions of basolateral complex of amygdala (BLA), cause impairments in many versions of the devaluation task (Balleine, Killcross, & Dickinson, 2003;Blundell, Hall & Killcross, 2003;Hatfield, Han, Conley, Gallagher, & Holland, 1996;Malkova, Gaffan & Murray, 1997). Several laboratories have also found that bilateral orbitofrontal cortex (OFC) lesions (Gallagher, McMahan & Schoenbaum, 1999;Izquierdo, Suda & Murray, 2004; but see Ostlund & Balleine, 2007) or lesions that disconnect amygdala and OFC (Baxter, Parker, Lindner, Izquierdo, & Murray, 2000) impair performance in devaluation tasks. NIH-PA Author ManuscriptNIH-PA Author Manuscript NIH-PA Author ManuscriptAnother brain area that may be needed for devaluation is mediodorsal thalamus (MD). MD, BLA and OFC are all interconnected (Carmichael & Price, 1995;Ghashghaei & Barbas, 2002;Groenewegen, 1988;Ray & Price, 1992) and the three brain areas share s...

show abstract

Section: Discussionmentioning

confidence: 99%

A limited role for mediodorsal thalamus in devaluation tasks.

Pickens

2008

Behavioral Neuroscience

View full text Add to dashboard Cite

show abstract

“…OFC ϭ orbitofrontal cortex. stimulus-response habits (Balleine & Dickinson, 1998;Holland & Gallagher, 2004). This is typically studied in the context of a devaluation paradigm.…”

Section: Devaluationmentioning

confidence: 99%

Anatomy of a decision: Striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal.

Frank¹,

Claus²

2006

Psychological Review

525

505

View full text Add to dashboard Cite

The authors explore the division of labor between the basal ganglia-dopamine (BG-DA) system and the orbitofrontal cortex (OFC) in decision making. They show that a primitive neural network model of the BG-DA system slowly learns to make decisions on the basis of the relative probability of rewards but is not as sensitive to (a) recency or (b) the value of specific rewards. An augmented model that explores BG-OFC interactions is more successful at estimating the true expected value of decisions and is faster at switching behavior when reinforcement contingencies change. In the augmented model, OFC areas exert top-down control on the BG and premotor areas by representing reinforcement magnitudes in working memory. The model successfully captures patterns of behavior resulting from OFC damage in decision making, reversal learning, and devaluation paradigms and makes additional predictions for the underlying source of these deficits.Keywords: decision making, neural network, basal ganglia, orbitofrontal cortex, reinforcement learning What enables humans to make choices that lead to long-term gains, even when having to incur short-term losses? Such decisionmaking skills depend on the processes of action selection (choosing between one of several possible responses) and reinforcement learning (modifying the likelihood of selecting a given response on the basis of experienced consequences). Although all mammals can learn to associate their actions with consequences, humans are particularly advanced in their ability to flexibly modify the relative reinforcement values of alternative choices to select the most adaptive behavior in a particular behavioral, spatial, and temporal context.The behavioral and cognitive neurosciences have identified two neural systems that are involved in such adaptive behavior. On the one hand, the basal ganglia (BG) and the neuromodulator dopamine (DA) are thought to participate in both action selection and reinforcement learning

show abstract

“…Animals learn act outcomes and select their acts based on the motivational value of these outcomes (reviews in Thistlethwaite, 1951;MacCorquodale and Meehl, 1954;Mackintosh 1974;Dennett 1978;Dickinson 1980;Dickinson 1994;Dickinson and Balleine 1994;Balleine and Dickinson 1998). Such goal-directed behavior is termed "planning" in reinforcement learning studies, or "cognition" in animal learning studies (Craik 1943;Sutton andBarto, 1981, 1998;Sutton and Pinette 1985;Dickinson 1994).…”

Section: Planning and Sensorimotor Learning Animal Learningmentioning

confidence: 99%

“…Furthermore, animals seem to use these predictions to select their acts (Sutton and Pinette, 1985;Sutton and Barto 1981;Dickinson and Balleine 1994;Balleine and Dickinson 1998). This insight led to the implementation of neural network architectures in which an internal model, serving as the Critic, transiently influences the Actor to elicit acts (Sutton and Barto, 1981).…”

Section: Animal Learning Modelsmentioning

confidence: 99%

Modeling functions of striatal dopamine modulation in learning and planning

Suri¹,

Bargas

Arbib³

2001

Neuroscience

View full text Add to dashboard Cite

The activity of midbrain dopamine neurons is strikingly similar to the reward prediction error of TD reinforcement learning models. Experimental evidence and simulation studies suggest that dopamine neuron activity serves as an effective reinforcement signal for learning of sensorimotor associations in striatal matrisomes.In the current study, we simulate dopamine neuron activity with the Extended TD model (Suri and Schultz, submitted) and examine the influence of this signal on medium spiny neurons in striatal matrisomes. This model includes transient membrane effects of dopamine, dopamine-dependent longterm adaptations of corticostriatal transmission, and rhythmic fluctuations of the membrane potential between an elevated "up-state" and a hyperpolarized "down-state." The most dominant activity in the striatal matrisomes elicits behaviors via projections from the basal ganglia to the thalamus and the cortex. This "standard model" performs successfully when tested for sensorimotor learning and goaldirected behavior (planning). To investigate the contributions of these model assumptions to learning and planning, we test the performance of several model variants that lack one of these mechanisms. These simulations show that the adaptation of the dopamine-like signal is necessary for planning and for sensorimotor learning. Lack of dopamine-like novelty responses decreases the number of exploratory acts, which deteriorates planning capabilities. Sensorimotor learning requires dopamine-dependent long-term adaptation of corticostriatal transmission. The model loses its planning capabilities if the dopamine-like signal is simulated with the original TD model. The capability for planning is improved by transient dopamine membrane effects, dopamine-dependent long-term effects on corticostriatal transmission, dopamine-and input-dependent influences on the durations of membrane potential fluctuations, and manipulations that prolong the reaction time of the model. These simulation results suggest that striatal dopamine is important for sensorimotor learning, exploration, and planning.

show abstract

Goal-directed instrumental action: contingency and incentive learning and their cortical substrates

Cited by 1,367 publications

References 41 publications

A limited role for mediodorsal thalamus in devaluation tasks.

A limited role for mediodorsal thalamus in devaluation tasks.

Anatomy of a decision: Striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal.

Modeling functions of striatal dopamine modulation in learning and planning

Contact Info

Product

Resources

About