Disruption of Dorsolateral Prefrontal Cortex Decreases Model-Based in Favor of Model-free Control in Humans

Self Cite

189

203

Behavioral and neural evidence reveal a prospective goal-directed decision process that relies on mental simulation of the environment, and a retrospective habitual process that caches returns previously garnered from available choices. Artificial systems combine the two by simulating the environment up to some depth and then exploiting habitual values as proxies for consequences that may arise in the further future. Using a three-step task, we provide evidence that human subjects use such a normative plan-until-habit strategy, implying a spectrum of approaches that interpolates between habitual and goal-directed responding. We found that increasing time pressure led to shallower goal-directed planning, suggesting that a speed-accuracy tradeoff controls the depth of planning with deeper search leading to more accurate evaluation, at the cost of slower decision-making. We conclude that subjects integrate habit-based cached values directly into goal-directed evaluations in a normative manner.planning | habit | reinforcement learning | speed/accuracy tradeoff | tree-based evaluation B ehavioral and neural evidence suggest that the brain uses distinct goal-directed and habitual systems for decisionmaking (1-5). A goal-directed system exploits an individual's model, i.e., their knowledge of environmental dynamics, to simulate the consequences that will likely follow a choice (6) (Fig. 1A). Such evaluations, which assess a decision-tree expanding into the future to estimate the total reward, adapt flexibly to changes in environmental dynamics or the values of outcomes. Evaluating deep trees, however, is computationally expensive (in terms of time, working memory, metabolic energy, etc.) and potentially error-prone. By contrast, the habitual system simply caches the rewards received on previous trials conditional on the choice (Fig. 1C) without a representational characterization of the environment (hence being called "model-free") (6, 7). This process hinders adaptation to changes in the environment, but has advantageous computational simplicity. Previous studies show distinct behavioral and neurobiological signatures of both systems (8-18). Furthermore, consistent with the theoretical strengths and weaknesses of each system (2, 19), different experimental conditions influence the relative contributions of the two systems in controlling behavior according to their respective competencies (20)(21)(22)(23).Here, we suggest that individuals, rather than simply showing greater reliance on the more competent system in each condition, combine the relative strengths of the two systems in a normative manner by integrating habit-based cached values directly into goaldirected evaluations. Specifically, we propose that given available resources (time, working memory, etc.), individuals decide the depth k up to which they can afford full forward simulations and use cached habitual values thereafter. That is, individuals compute the value of a choice by adding the first k rewards, predicted by the explicit simulation, to the value...

supporting

confidence: 63%

mentioning

confidence: 89%

Adaptive integration of habits into depth-limited planning defines a habitual-goal–directed spectrum

Keramati

Smittenaar

Dolan

et al. 2016

Self Cite

189

203

“…In a similar vein, dopamine is implicated in a modulation of PFC maintenance processes via a gating of cortical gain, rendering coding of relevant environmental information more robust against noise (11,26,27). Indeed, the importance of lateral PFC for model-based inference is supported by findings that theta-burst transcranial magnetic stimulation compromises model-based control in humans (28). Our analysis of second-stage reaction times, which were affected by the state transition matrix, showed that a response time difference for rare versus common states was positively related to a bias toward more model-based choices.…”

Section: Discussionmentioning

confidence: 97%

Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making

Deserno

Huys

Boehme

et al. 2015

224

212

Dual system theories suggest that behavioral control is parsed between a deliberative "model-based" and a more reflexive "modelfree" system. A balance of control exerted by these systems is thought to be related to dopamine neurotransmission. However, in the absence of direct measures of human dopamine, it remains unknown whether this reflects a quantitative relation with dopamine either in the striatum or other brain areas. Using a sequential decision task performed during functional magnetic resonance imaging, combined with striatal measures of dopamine using [ 18 F]DOPA positron emission tomography, we show that higher presynaptic ventral striatal dopamine levels were associated with a behavioral bias toward more model-based control. Higher presynaptic dopamine in ventral striatum was associated with greater coding of model-based signatures in lateral prefrontal cortex and diminished coding of model-free prediction errors in ventral striatum. Thus, interindividual variability in ventral striatal presynaptic dopamine reflects a balance in the behavioral expression and the neural signatures of model-free and model-based control. Our data provide a novel perspective on how alterations in presynaptic dopamine levels might be accompanied by a disruption of behavioral control as observed in aging or neuropsychiatric diseases such as schizophrenia and addiction.dopamine | decision making | reinforcement learning | PET | fMRI H uman choice behavior is influenced by both habitual and goal-directed systems (1). For example, having enjoyed a delicious dinner makes another subsequent visit to the same restaurant more likely. Upon returning at a later point, another visit could happen reflexively when walking past the restaurant, or alternatively be planned and involve reflection, for instance, by checking recent customer reviews to bolster against possible changes. These two decision modes differ fundamentally in terms of their control over actions and associated outcome consequences. Reflexive habitual preferences are retrospective and arise from a slow accumulation of rewards via iterative updating of expectations (2), for example by repeating dinner at the same place after having previously enjoyed tasty food there. In contrast, goal-directed behavior requires a prospective consideration of future outcomes associated with a set of actions (3). For example, knowledge that the chef has changed and subsequent reviews have been less good should reduce one's expectations. Thus, in the face of such change, a goaldirected system can adapt quickly, whereas a habitual system needs to experience an actual outcome before it can alter behavior in an adaptive manner (4). This dual-system theory has been formalized within computational models of learning that update expectations based on past rewards ("model-free") or map possible actions to their potential outcomes ("model-based") (5). There is evidence that model-based learning signals during the acquisition of task structure are encoded within prefrontal-parietal cortices, where...

“…In the most salient work along these lines, reward-based tree search has been conceptualized in terms provided by modelbased reinforcement learning, a computational framework in which reward-based decisions are based on an explicit model of the choice problem, a "cognitive map" of the decision tree itself (11). Under this rubric, recent work has illuminated several aspects of rewardbased tree search, providing an indication of how representations of decision problems are acquired and updated (12)(13)(14), where in the brain relevant quantities (e.g., cumulative rewards) are represented (15-17), and how model-based decision making interacts with simpler, habit-based choice mechanisms (15,(18)(19)(20)(21)(22).Despite such advances, however, comparatively little progress has so far been made toward characterizing the concrete process by which model-based decisions are reached, that is, the actual procedure through which a representation of the decision problem is translated into a choice (9,10,23). This situation contrasts sharply with what one finds in the literature on simple choice, where a number of detailed process models have been proposed.…”

mentioning

confidence: 99%

mentioning

confidence: 99%

Evidence integration in model-based tree search

Solway

Botvinick

2015

Research on the dynamics of reward-based, goal-directed decision making has largely focused on simple choice, where participants decide among a set of unitary, mutually exclusive options. Recent work suggests that the deliberation process underlying simple choice can be understood in terms of evidence integration: Noisy evidence in favor of each option accrues over time, until the evidence in favor of one option is significantly greater than the rest. However, real-life decisions often involve not one, but several steps of action, requiring a consideration of cumulative rewards and a sensitivity to recursive decision structure. We present results from two experiments that leveraged techniques previously applied to simple choice to shed light on the deliberation process underlying multistep choice. We interpret the results from these experiments in terms of a new computational model, which extends the evidence accumulation perspective to multiple steps of action.reward-based decision making | drift-diffusion model | reinforcement learning I magine a customer standing at the counter in an ice cream shop, deliberating among the available flavors. Such a scenario exemplifies "simple choice," a decision situation in which the objective is to select among a set of individual, immediate outcomes, each carrying a different reward. Simple choice, in this sense, has provided a convenient focus for a great deal of work in behavioral economics and decision neuroscience (1-5). However, it would be an obvious mistake to treat it as an exhaustive model of rewardbased decision making. The decisions that arise in everyday life are of course often more complicated. One important difference, among others, is that everyday decisions tend to involve sequences of actions and outcomes.As an illustration, let us return to the ice cream customer, picturing him at a point slightly earlier in the day, exiting his home in quest of something sweet. Upon reaching the sidewalk, he faces a decision between heading left toward the ice cream shop, or heading right toward a frozen yogurt shop. If he wishes to fully evaluate the relative appeal of these two options, he must answer a second set of questions: Which flavor would he choose in each shop? Furthermore, it may be relevant for him to consider more immediate consequences of the left-right decision. For example, the leftward path might pass by a bank, allowing him to deposit a check along his way, whereas the rightward path might lead by the post office, giving him the opportunity to mail a package.Rather than selecting among individual and immediate outcomes, the decision maker in this scenario finds himself at the root of a decision tree (Fig. 1A), with nodes corresponding to value-laden outcomes or states, and edges corresponding to choiceinduced state transitions. Deciding among immediate actions, even at the first branch point, requires a consideration of all of the paths that unfold below. Decision making thus assumes the form of reward-based tree search (6-10).Note that decision makin...