When to retrieve and encode episodic memories: a neural network model of hippocampal-cortical interaction

Abstract

When should episodic memories be stored and retrieved to support event understanding? Traditional list-learning memory experiments make it obvious when to store and retrieve memories, but it is less obvious when to do this in naturalistic settings. To address this question, we trained a memory-augmented neural network to predict upcoming events, in an environment where situations (sets of parameters governing transitions between events) sometimes reoccurred. The model was allowed to learn a policy for when to consult episodic memory, and we explored how this learned policy varied as a function of the task environment. We found that the learned retrieval policy is shaped by internal uncertainty about upcoming events, the level of penalty associated with incorrect predictions, the confusability of stored memories, the presence of a “familiarity signal” indicating the availability of relevant memories, and the presence of statistical regularities (prototypical events). With regard to encoding policy, we found that selectively storing episodic memories at the end of an event (but not mid-event) leads to better subsequent event prediction performance and less incorrect recall. Additionally, we found that the model can integrate information over long timescales even without the hippocampus; it can link information over many event segments via episodic memory; and it shows classic schema-consistent memory effects when the upcoming time point has a prototypical event. Overall, these modeling results provide a normative explanation of several existing behavioral and neuroimaging findings regarding the use of episodic memory in naturalistic settings, and lead to a wide range of testable predictions.

A content-based representational scaffold for naturalistic event memories

“…This suggests that representational timescales may be another important factor which differs across regions, broadly consistent with previous work suggesting relatively faster and slower pattern changes during temporally extended stimuli such as movies (Chen et al, 2015;Hasson et al, 2008;Hasson et al, 2015;Honey et al, 2012;Lerner et al, 2011). With regards to HPC, however, our result of relatively higher pattern similarity at event onset and offset than mid-event aligns with the idea of HPC being particularly important for encoding information at event boundaries (Ben-Yakov et al, 2013;Lu et al, 2020). Lu et al used a computational modeling approach to find that selectively storing information at event boundaries (but not mid-event) was beneficial to storing and later retrieving that information (Lu et al, 2020).…”

“…With regards to HPC, however, our result of relatively higher pattern similarity at event onset and offset than mid-event aligns with the idea of HPC being particularly important for encoding information at event boundaries (Ben-Yakov et al, 2013;Lu et al, 2020). Lu et al used a computational modeling approach to find that selectively storing information at event boundaries (but not mid-event) was beneficial to storing and later retrieving that information (Lu et al, 2020). Relatedly, a recent study by Cohn-Sheehy and colleagues suggests that event boundaries may act not only as moments in which an event transition occurs, but moments in which related information can be integrated across distinct experiences (Cohn-Sheehy et al, 2020).…”

“…In order to simplify analyses, we chose to bin event epochs. Given prior evidence for the importance of HPC representations at event boundaries (Ben-Yakov et al, 2013;Lu et al, 2020), 40-second events were binned into event Onset (the first 8 TRs), event Offset (the last 8 TRs), and Mid event (the intermediate 17 TRs). For each ROI, each epoch was contrasted against its mean pattern similarity value across all events, across all runs (i.e., the grand mean).…”

“…Given that different ROIs responded to different conditions, we chose to compare only same-event neural patterns across runs for this analysis, as this is the only condition in which all ROIs reliably showed pattern similarity increases across participants. Furthermore, given evidence for distinct mechanisms of hippocampal encoding at event boundaries (Ben-Yakov et al, 2013;Lu et al, 2020), we binned the 33-TR time series into event Onset (the first 8 TRs), event Offset (the final 8 TRs), and Mid event timepoints (the intermediate 17 TRs). This additionally simplified analyses of the data while nonetheless allowing for evaluation of a temporally-evolving change in pattern similarity.…”

Predicting memory from the network structure of naturalistic events

“…Consideration of the effects of inter-event structure on real-world memory may benefit practical applications such as the development of memory interventions for clinical and healthy aging populations 65 or promoting learning in educational settings [66][67][68] . In addition, our work demonstrates that holistic metrics which capture the interrelations of events within episodes may be important to incorporate into models of learning and comprehension, especially as these models grow in their sophistication and power to explain complex experiences in the real world 69,70 .…”