Spatiotemporal neural characterization of prediction error valence and surprise during reward learning in humans

Abstract

Reward learning depends on accurate reward associations with potential choices. These associations can be attained with reinforcement learning mechanisms using a reward prediction error (RPE) signal (the difference between actual and expected rewards) for updating future reward expectations. Despite an extensive body of literature on the influence of RPE on learning, little has been done to investigate the potentially separate contributions of RPE valence (positive or negative) and surprise (absolute degree of deviation from expectations). Here, we coupled single-trial electroencephalography with simultaneously acquired fMRI, during a probabilistic reversal-learning task, to offer evidence of temporally overlapping but largely distinct spatial representations of RPE valence and surprise. Electrophysiological variability in RPE valence correlated with activity in regions of the human reward network promoting approach or avoidance learning. Electrophysiological variability in RPE surprise correlated primarily with activity in regions of the human attentional network controlling the speed of learning. Crucially, despite the largely separate spatial extend of these representations our EEG-informed fMRI approach uniquely revealed a linear superposition of the two RPE components in a smaller network encompassing visuo-mnemonic and reward areas. Activity in this network was further predictive of stimulus value updating indicating a comparable contribution of both signals to reward learning.

Separate neural representations of prediction error valence and surprise: Evidence from an fMRI meta‐analysis

“…Subsequently, to formally test for the overlap between all three RPE components and identify potential regions integrating valence and surprise either into a signed RPE representation or a linear superposition of the two signals (Fouragnan et al, ), we performed a conjunction analysis between the valence (pattern A), the surprise (pattern B) and signed RPE (pattern C) signals. We summarize our conjunction results in Figure , which revealed a major overlap between all activations associated with signed RPE and each of the other two RPE representations in the central part of the STR.…”

“…First, these studies have revealed two temporally specific EEG components discriminating between positive and negative RPEs peaking around 220 and 300 ms, respectively, largely consistent with the timing of the feedback‐related negativity and feedback‐related positivity ERP components (Cohen, Elger, & Ranganath, ; Hajcak, Moser, Holroyd, & Simons, ; Yeung & Sanfey, ). Additionally, the studies also revealed a late unsigned RPE component which overlaps temporally with the late valence signal (Philiastides et al, ) but appears in a largely separate and distributed neural network (Fouragnan et al, ).…”

“…In contrast, it has also been suggested that a surprise system can independently monitor unexpected information and act as a learning signal that allows better predictions of upcoming events, and help plan appropriate behavioural adjustments (Dayan & Balleine, ; Fouragnan et al, ; Kolling, Behrens, Mars, & Rushworth, ; Wittmann et al, ). In particular, some studies suggest that the aINS receives information related to surprise and direct modulation from the dSTR providing crucial information for behavioural adjustment (Menon & Levitin, ).…”

“…In this framework, RPE is a signed quantity and learning is driven by two separate components of the RPE signal: its valence (i.e., the sign of the RPE, representing whether an outcome is better [+] or worse [−] than expected) and its surprise (i.e., the modulus of the RPE, representing the degree [high or low] of deviation from expectations). Whereas the valence informs an agent whether to reinforce or extinguish a certain behaviour (Fouragnan, Retzler, Mullinger, & Philiastides, ; Fouragnan, Queirazza, Retzler, Mullinger, & Philiastides, ; Frank, Seeberger, & O'reilly, ), the surprise component determines the extent to which the strength of association between outcome and expectations needs to be adjusted (Collins & Frank, ; Niv et al, ; den Ouden, Kok, & de Lange, ).…”

“…Additionally, human electroencephalography (EEG) studies, attempting to offer a temporal account of the cortical dynamics associated with RPE processing, did not find a systematic monotonic response profile consistent with a single RPE representation but instead offered evidence suggestive of separate representations for valence and surprise at the macroscopic level of responses recorded on the scalp. Specifically, multiple recent EEG studies combining model‐based RPE estimates with single‐trial analysis of the EEG revealed an early outcome stage reflecting a purely categorical valence signal and a later processing stage reflecting separate representations for valence and surprise (Fouragnan et al, ; Philiastides, Biele, Vavatzanidis, Kazzer, & Heekeren, ). These later valence and surprise signals appeared in spatially distinct but temporally overlapping neural signatures.…”