Reward prediction errors create event boundaries in memory

Abstract

We remember when things change. Particularly salient are experiences where there is a change in rewards, eliciting reward prediction errors (RPEs). This feature of memory may be useful because it can help us find greater rewards and avoid lesser ones in the future. How do RPEs influence our memory of those experiences? One idea is that this signal directly enhances the encoding of memory. Another, not mutually exclusive, idea is that the RPE signals a deeper change in the environment, and leads to the mnemonic separation of subsequent experiences from what came before, thereby creating a new latent context and a more separate memory trace. We tested this in four experiments in which participants learned to predict rewards associated with a series of images within visually-distinct "rooms." High magnitude RPEs indicated a change in the underlying distribution of rewards. To test whether these large RPEs created a new latent context, we first assessed recognition priming for sequential pairs that contained or did not contain a high-RPE event, as well as out-of-sequence pairs (Exp. 1: n=27 & Exp. 2: n=83). We found evidence of recognition priming for both sequential pair types, including the pair with the high-RPE event, indicating that the high-RPE event is bound to its predecessor in memory. Given that high-RPE events are themselves preferentially remembered (Rouhani et al, 2018), we next tested recognition priming for pairs that had one item in between them (i.e. the pairs were either across a high-RPE event or not), where none of the tested items were high-RPE items (Exp. 3: n=85). Here, sequential pairs across a high-RPE no longer showed recognition priming whereas pairs within the same latent reward state did, providing initial evidence for an RPE-modulated event boundary. We then investigated whether RPE event boundaries disrupt temporal memory of those events (Exp. 4). After reward learning, we asked participants to order and estimate the distance between two events that had either included a high-RPE event between them, or not. We found (n=49) and replicated (n=77) worse sequence memory for events across a high-RPE event. Altogether, these findings demonstrate that high-RPE events are both more strongly encoded and act as event boundaries that interrupt the sequential integration of events. We captured these effects in a variant of the Context Maintenance and Retrieval model (CMR; Polyn, Norman & Kahana, 2009), modified to incorporate RPEs into the encoding process.

Pupil-linked arousal signals track the temporal organization of events in memory

“…6a, b). Increasing evidence suggests that context shifts modulate memory for and attention to information presented at boundaries 16,18,47 . Because boundary information may be processed differently in attention and memory than same-context information, we reasoned that the "boundary-ness" of both the context shift and the following item may relate to a spike in arousal (captured by pupil dilations to both the tone and its following item).…”

“…Boundary salience does not relate to pupil size or memory. Prior work suggests that the magnitude of prediction errors, which have been theorized to induce event boundaries, scales with event segmentation effects in memory 18 . Moreover, prediction errors are associated with activation of brainstem arousal systems.…”

“…However, when the surrounding context changes, such as entering a new room or being interrupted by a phone call, people tend to perceive an event boundary that defines the end of the current event and the beginning of a new one 7,8 . Importantly, these event boundaries have a significant impact on how we remember experiences later on by promoting more separated memory representations 1,4,5,[9][10][11][12][13][14][15][16][17][18] . Thus, temporal stability and change in an unfolding context, including fluctuations in an individual's surroundings or mental state, help to form a mental timeline of discrete episodic memories.…”

“…For instance, when individuals are presented with two items from a recent experience, memory for the order of those items is relatively impaired if there was an intervening context change compared to instances where the items were encountered in the same context [12][13][14][15][16][17][18][19] . Likewise, item pairs are also remembered as having occurred farther apart in time if they spanned an event boundary compared to items that had been encountered in the same context 1,4,18 . Event boundaries can also influence nontemporal aspects of episodic memory, such as enhancing associative memory for an item and any co-incident contextual information (e.g., background color 16,20 ).…”

“…Many salient environmental changes, such as hearing an unexpected sound, can activate central arousal systems that regulate ongoing attention and memory processes across the brain [27][28][29] . Furthermore, arousal signals mediate some of the same cognitive processes that are thought to be triggered by event boundaries, including cognitive control, prediction errors, and attention re-orienting 7,18,[29][30][31] . Of relevance to the current study, emerging evidence also suggests that arousal responses are sensitive to the structure of temporally extended experiences 32,33 .…”