Modelling Individual Differences in the Form of Pavlovian Conditioned Approach Responses: A Dual Learning Systems Approach with Factored Representations

Abstract

Reinforcement Learning has greatly influenced models of conditioning, providing powerful explanations of acquired behaviour and underlying physiological observations. However, in recent autoshaping experiments in rats, variation in the form of Pavlovian conditioned responses (CRs) and associated dopamine activity, have questioned the classical hypothesis that phasic dopamine activity corresponds to a reward prediction error-like signal arising from a classical Model-Free system, necessary for Pavlovian conditioning. Over the course of Pavlovian conditioning using food as the unconditioned stimulus (US), some rats (sign-trackers) come to approach and engage the conditioned stimulus (CS) itself – a lever – more and more avidly, whereas other rats (goal-trackers) learn to approach the location of food delivery upon CS presentation. Importantly, although both sign-trackers and goal-trackers learn the CS-US association equally well, only in sign-trackers does phasic dopamine activity show classical reward prediction error-like bursts. Furthermore, neither the acquisition nor the expression of a goal-tracking CR is dopamine-dependent. Here we present a computational model that can account for such individual variations. We show that a combination of a Model-Based system and a revised Model-Free system can account for the development of distinct CRs in rats. Moreover, we show that revising a classical Model-Free system to individually process stimuli by using factored representations can explain why classical dopaminergic patterns may be observed for some rats and not for others depending on the CR they develop. In addition, the model can account for other behavioural and pharmacological results obtained using the same, or similar, autoshaping procedures. Finally, the model makes it possible to draw a set of experimental predictions that may be verified in a modified experimental protocol. We suggest that further investigation of factored representations in computational neuroscience studies may be useful.

Isolating the incentive salience of reward-associated stimuli: value, choice, and persistence

“…From this point of view, individual differences in rate of response decay under omission schedules and rate of discounting during lever choice express the relative dominance of S -R learning over A-O learning. Indeed, this approach has recently been extended to individual differences in PCA performance using the cohort method, where sign-trackers have larger v values and goal-trackers have lower v values, leading to behavior governed by model-free and model-based systems, respectively (Lesaint et al 2014). Collectively, the data suggest that sign-tracking in response to a lever is controlled by Pavlovian S-R relationships, while goal-tracking in response to a tone is controlled by instrumental A-O relationships, making the function of the tone in reference to goaltracking more akin to a discriminative stimulus that modulates A-O contingencies, rather than a Pavlovian CS.…”

“…Given that the two proposed neurobehavioral systems thought to govern sign-and goal-tracking response types are considered independent but parallel processes within individuals (Dayan et al 2006;Clark et al 2012;Dayan and Berridge 2014;Huys et al 2014), we reasoned that specifically engaging these two systems within a single individual would better isolate and compare the functional disparities between each. This 2-CS PCA task would also be useful for studying the development of the proposed arbitration process thought to determine the relative dominance of one system over the other (Dayan et al 2006;Huys et al 2014;Lesaint et al 2014), a process that would be particularly difficult to assess using between-subject cohort designs.…”

The role of learning-related dopamine signals in addiction vulnerability

“…This has consequences for learning accounts of addiction as some learning tendencies appear to confer vulnerability towards developing addiction. In this part, we first present the data on individual differences in Pavlovian responding in some detail (mainly reiterating the findings of Flagel et al 2011b), then discuss its interpretation in terms of incentive salience (Berridge and Robinson, 1998;Berridge, 2004Berridge, , 2007Saunders and Robinson, 2012), and finally put forth a hypothesis that proposes a connection between the propensity to assign incentive salience and the propensity to employ model-free learning (McClure et al, 2003a;Huys et al, 2013b;Lesaint et al, 2013;Dayan and Berridge, 2013).…”

“…We now consider the hypothesis that model-free and model-based learning may at least partially map onto sign-and goal-tracking behaviour, respectively (Lesaint et al, 2013;Huys et al, 2013b). In Pavlovian conditioning experiments, reward delivery is independent of the animals' behaviour.…”

“…Goal-trackers focus on the goal as soon as the sign appears, but they also focus on the goal during the ITI (a time when the sign is not present) when no food is presented there. This may lead to keeping the model-free values of both the goal and the CS near zero (though ITI head-entries into the food-cup do not differ between goal-and sign-trackers; see Lesaint et al 2013 for a detailed discussion).…”