Geometric models reveal behavioural and neural signatures of transforming experiences into memories

Abstract

The mental contexts in which we interpret experiences are often person-specific, even when the experiences themselves are shared. We developed a geometric framework for mathematically characterizing the subjective conceptual content of dynamic naturalistic experiences. We model experiences and memories as trajectories through word embedding spaces whose coordinates reflect the universe of thoughts under consideration. Memory encoding can then be modeled as geometrically preserving or distorting the shape of the original experience. We applied our approach to data collected as participants watched and verbally recounted a television episode while undergoing functional neuroimaging. Participants' recountings all preserved coarse spatial properties (essential narrative elements), but not fine spatial scale (low-level) details, of the episode's trajectory. We also identified networks of brain structures sensitive to these trajectory shapes. Our work provides insights into how we preserve and distort our ongoing experiences when we encode them into episodic memories.

Predicting memory from the network structure of naturalistic events

“…Different colors indicate different movies. Consistent with a prior study 26 , the overall configuration of the recall trajectories was similar to that of the movie annotation trajectory. The recall trajectories were also similar across subjects, although the number of movies recalled and the number of utterances made varied across subjects.…”

“…In accordance with this view, we observed event-specific neural activation patterns in DMN areas during recall, and representational similarity analysis revealed that the relational structure of these neural event patterns could be predicted by human-generated descriptions of the movie and by recall transcripts (for a similar approach, see ref. 26 ). Critically, higher semantic centrality predicted greater activation (Supplementary Figure 7b) and between-subject pattern convergence in PMC (Figure 4e), neural signatures of stronger and more accurate recall of episodic details 17,29,47 .…”

Building bridges: the hippocampus constructs narrative memories across distant events

“…Several studies have suggested that hippocampal-cortical interactions support the ability to continuously track information across multiple events within a narrative, as well as event boundaries (32,33,44,46,64). Recent findings from Heusser et al (65) suggest that the Posterior Medial and Anterior Temporal subnetworks of the default mode network (42,43) provide complementary support for the representation of events within larger narratives. Future studies should elucidate the link between cortical support for processing temporally extended narratives, and hippocampal support for integrating temporally distant events in memory.…”

“…Many studies have demonstrated cortical support for the representation of narrative structure (e.g. Aly et al, 2018;Baldassano et al, 2017;Chang et al, 2021;Chen et al, 2017;Heusser et al, 2021). For instance, Aly et al (2018) scanned participants while they repeatedly watched movie clips in which the order of events was intact, and other clips in which the order of events was scrambled so that it did not preserve the narrative structure.…”

Narratives: fMRI data for evaluating models of naturalistic language comprehension

“…For example, the Narratives data are particularly well-suited for evaluating models capturing linguistic content ranging from lower-level acoustic features 97–99 to higher-level semantic features 46,100–102 . More broadly, naturalistic data of this sort can be useful for evaluating shared information across subjects 103,104 , individual differences 76,105–107 , algorithms for functional alignment algorithms for functional alignment 80,108–112 , models of event segmentation and narrative context 113–117 , and functional network organization 118–121 . In the following, we describe the Narratives data collection and provide several perspectives on data quality.…”