Principal components analysis of reward prediction errors in a reinforcement learning task

Abstract

Models of reinforcement learning represent reward and punishment in terms of reward prediction errors (RPEs), quantitative signed terms describing the degree to which outcomes are better than expected (positive RPEs) or worse (negative RPEs). An electrophysiological component known as feedback related negativity (FRN) occurs at frontocentral sites 240-340 ms after feedback on whether a reward or punishment is obtained, and has been claimed to neurally encode an RPE. An outstanding question however, is whether the FRN is sensitive to the size of both positive RPEs and negative RPEs. Previous attempts to answer this question have examined the simple effects of RPE size for positive RPEs and negative RPEs separately. However, this methodology can be compromised by overlap from components coding for unsigned prediction error size, or "salience", which are sensitive to the absolute size of a prediction error but not its valence. In our study, positive and negative RPEs were parametrically modulated using both reward likelihood and magnitude, with principal components analysis used to separate out overlying components. This revealed a single RPE encoding component responsive to the size of positive RPEs, peaking at ~330 ms, and occupying the delta frequency band. Other components responsive to unsigned prediction error size were shown, but no component sensitive to negative RPE size was found.

Temporal dissociation of salience and prediction error responses to appetitive and aversive taste

“…It is important to note that the main effect of outcome here drove the results reported above for the traditional difference-wave analysis. As per previous suggestions (Holroyd, 2004;Sambrook & Goslin, 2015b), the early response to motivational salience is likely related to the N2.…”

“…The feedback-related negativity (FRN) is thought to originate from dopaminergic projections to the anterior cingulate cortex (ACC), evident by the finding that it is modulated by dopamine agonists (Holroyd & Coles, 2002;Santesso et al, 2009;Walsh & Anderson, 2012) and combined EEG-fMRI work (Hauser et al, 2014). The dominant theory contends that the FRN expresses utility prediction error, but recent studies provided evidence that it expresses salience (Garofalo, Maier, & di Pellegrino, 2014;Hauser et al, 2014;Huang & Yu, 2014;Pfabigan et al, 2015;Sambrook & Goslin, 2015b;Talmi, Atkinson, & El-Deredy, 2013). These studies showed that the FRN reflects a negative deflection when any outcome-appetitive or aversive-is unexpectedly omitted, as proposed by the predicted response outcome (PRO) model (Alexander & Brown, 2011) and in line with an interpretation of the FRN as expressing motivational salience rather than utility prediction error signal.…”

Temporal dissociation of salience and prediction error responses to appetitive and aversive taste

“…Following the logic of the axiomatic model (Caplin & Dean, 2008;Rutledge et al, 2010) we were particularly interested in signal that differentiated aversive and appetitive taste and taste omission cues more strongly when they were unexpected. For both appetitive and aversive taste, we observed continued expression of salience (Sambrook & Goslin, 2015b) across the entire 200-380ms time window. There was also an expression of aversive but not reward prediction error at the latency most characteristic of the FRN, peaking at 285ms (Holroyd, 2004;Sambrook & Goslin, 2015a).…”

“…The feedback-related negativity (FRN) is thought to originate from dopaminergic projections to the anterior cingulate cortex (ACC), evident by the finding that it is modulated by dopamine agonists (Holroyd & Coles, 2002;Santesso et al, 2009;Walsh & Anderson, 2012) and combined EEG-fMRI work (Hauser et al, 2014) . The dominant theory contends that the feedback-related negativity (FRN) expresses utility prediction error, but recent studies provided evidence that it expresses salience (Garofalo, Maier, & di Pellegrino, 2014;Hauser et al, 2014;Huang & Yu, 2014;Pfabigan et al, 2015;Sambrook & Goslin, 2015b;Talmi, Atkinson, & El-Deredy, 2013).…”

Expectancy effects in feedback processing are explained primarily by time-frequency delta not theta

“…Cavanagh (2015) showed that delta activity is sensitive to positive reward prediction errors (+RPEs), or unexpected gains. Sambrook and Goslin (2016) demonstrated that theta was sensitive to negative reward prediction errors (-RPEs), or unexpected losses, and delta was larger for +RPEs. Furthermore, there was an interaction between the size and valence of the RPE in delta, where delta was largest for high magnitude +RPEs.…”