Signed and unsigned reward prediction errors dynamically enhance learning and memory

Abstract

Memory helps guide behavior, but which experiences from the past are prioritized? Classic models of learning posit that events associated with unpredictable outcomes as well as, paradoxically, predictable outcomes, deploy more attention and learning for those events. Here, we test reinforcement learning and subsequent memory for those events, and treat signed and unsigned reward prediction errors (RPEs), experienced at the reward-predictive cue or reward outcome, as drivers of these two seemingly contradictory signals. By fitting reinforcement learning models to behavior, we find that both RPEs contribute to learning by modulating a dynamically changing learning rate. We further characterize the effects of these RPE signals on memory, and show that both signed and unsigned RPEs enhance memory, in line with midbrain dopamine and locus-coeruleus modulation of hippocampal plasticity, thereby reconciling separate findings in the literature.

Mixing memory and desire: How memory reactivation supports deliberative decision‐making

“…These results suggest that memories about specific rewarding events are successfully encoded and then subsequently reactivated upon a second encounter, consistent with the idea of evidence arising from discrete packets, and with an evaluation function that is predicated on the value experienced in that previous episode, rather than one computed anew. However, these data could also be consistent with separate effects of positive reward prediction errors on choice and memory (Jang et al, 2019;Rouhani & Niv, 2021) . The question of whether memory sampling requires explicit recollection at the time of choice remains an area of active interest.…”

Abrupt hippocampal remapping signals resolution of memory interference

“…Although speculative, it is possible that when a given scene activated the “wrong” object association (at LR-1), this actively triggered a correction in favor of the target object association that supported learning. This account is consistent with evidence that prediction errors can powerfully drive episodic memory 45 , 46 as well as differentiation of hippocampal activity patterns 19 . More broadly—and consistent with our findings, in general—prediction errors may induce abrupt state changes in the hippocampus that facilitate the separation of episodic memories 47 .…”

Surprise: a unified theory and experimental predictions

“…Antony et al, 2021 showed that a similar measure of surprise plays a role in the segmentation of memory and correlates with pupil dilation and the activity of dopamine-related regions of the brain in fMRI. The influence of surprise on memory has also been studied in reward-driven experiments where it has been shown that high unsigned reward prediction errors increase the memorability of task-independent stimuli (Rouhani & Niv, 2021;Rouhani et al, 2018); in such settings, reward can be defined as a component of the observation y t , and the unsigned reward prediction error can be seen as the absolute error surprise (c.f. Eq.…”

“…(Gottlieb & Oudeyer, 2018), 'how can we formulate other surprise theories like Palm's theory of surprise (Palm, 2012) or the theory of surprise as a compression measure (Schmidhuber, 2010) in our mathematical framework? ', and 'how do different surprise measures contribute to formation (Rouhani & Niv, 2021), segmentation (Antony et al, 2021;Rouhani et al, 2020), and modification (Gershman et al, 2017;Sinclair & Barense, 2018) of memory? '.…”