Model-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation

Abstract

Evidence supports at least two methods for learning about reward and punishment and making predictions for guiding actions. One method, called model-free, progressively acquires cached estimates of the long-run values of circumstances and actions from retrospective experience. The other method, called model-based, uses representations of the environment, expectations and prospective calculations to make cognitive predictions of future value. Extensive attention has been paid to both methods in computational analyses of instrumental learning. By contrast, although a full computational analysis has been lacking, Pavlovian learning and prediction has typically been presumed to be solely model-free. Here, we revise that presumption and review compelling evidence from Pavlovian revaluation experiments showing that Pavlovian predictions can involve their own form of model-based evaluation. In model-based Pavlovian evaluation, prevailing states of the body and brain influence value computations, and thereby produce powerful incentive motivations that can sometimes be quite new. We consider the consequences of this revised Pavlovian view for the computational landscape of prediction, response and choice. We also revisit differences between Pavlovian and instrumental learning in the control of incentive motivation.

Dissociating contributions of ventral and dorsal striatum to reward learning

“…As the probability of reward receipt increased, BOLD signal in DLS and DMS also increased. At first glance, it is tempting to conclude that these two areas track the probability of reward and therefore code model-based learning (Daw et al 2005(Daw et al , 2011Dayan and Berridge 2014). However, after a closer examination, we argue that this is actually not the case.…”

“…Both at the beginning of the experiment and after the devaluation procedure, subjects were asked to rate the pleasantness of the two foods. The pleasantness reflects the hedonic impact of a food in the consummatory phase, i.e., subjective "liking" (Dayan and Berridge 2014). The authors found that the pleasantness rating of the devalued food following the selective satiation procedure decreased significantly more than that of the valued food.…”

“…Recent research employing the computational framework has greatly elucidated the underlying process of reward learning (Daw et al 2005(Daw et al , 2011Dayan and Berridge 2014). Under this framework, subjects predict the reward value of cues according to mathematical value functions.…”

“…Two major forms of reward learning that have been studied extensively in the past century are Pavlovian learning (or classical conditioning) and instrumental learning (or operant conditioning) (Dayan and Berridge 2014). In Pavlovian learning, subjects predict the reward value by a conditioned cue that occurs before it.…”

“…1B). More recently, Dayan and Berridge (2014) brought the model-free and model-based paradigm into Pavlovian learning. They suggested that DLS and the core of VS are involved in model-free Pavlovian learning, whereas DMS and the shell of VS are necessary for model-based Pavlovian learning (Fig.…”