Structured event memory: a neuro-symbolic model of event cognition

Abstract

Humans spontaneously organize a continuous experience into discrete events and use the learned structure of these events to generalize and organize memory. We introduce the Structured Event Memory (SEM) model of event cognition, which accounts for human abilities in event segmentation, memory, and generalization. SEM is derived from a probabilistic generative model of event dynamics defined over structured symbolic scenes. By embedding symbolic scene representations in a vector space and parametrizing the scene dynamics in this continuous space, SEM combines the advantages of structured and neural network approaches to high-level cognition. Using probabilistic reasoning over this generative model, SEM can infer event boundaries, learn event schemata, and use event knowledge to reconstruct past experience. We show that SEM can scale up to high-dimensional input spaces, producing human-like event segmentation for naturalistic video data, and accounts for a wide array of memory phenomena.

Behavioral, physiological, and neural signatures of surprise during naturalistic sports viewing

“…Our findings reveal that the popular activity of competitive sports viewing is a valuable model of naturalistic surprise, and our analyses of this task led to multiple novel behavioral and physiological discoveries in support of the tenets of event segmentation theory (EST) 9,16 . Namely, surprises appear to strongly drive the segmentation of internal event representations, indexed by increased subjective perception of event boundaries, increased pupil dilation, an increased likelihood of significant neural representational shifts (as measured using a Hidden Markov Model), and increased subsequent memory for events.…”

“…EST posits that the segmentation of ongoing experience into discrete events enhances memory for information near the event boundaries 2,59-61 . Given that surprise helps create event boundaries 9 and enhances memory in laboratory settings 2,3,5,62 , we asked how it and other factors described above predict long-term memory in our naturalistic paradigm.…”

“…Of course, our predictions are not always correct -when they are inaccurate, we experience surprise (i.e., unsigned prediction error [1][2][3][4][5]. Surprise is theorized to be critical for learning and memory 6,7 , updating our beliefs about the structure of the world 8 , and demarcating events in the continuous flow of time 9 . Moreover, although people typically prefer certainty about outcomes that are instrumental for survival 10 , in domains with noninstrumental information, such as narratives, music, and sports, people tend to prefer violations of their expectations [11][12][13] , suggesting that surprise is often a rewarding affective experience.…”

Constructing and Forgetting Temporal Context in the Human Cerebral Cortex

“…Gated neural networks have been widely applied in sequence learning models to capture long-range temporal dependencies (Hochreiter and Schmidhuber, 1997). Combining both the gated neural network approach and probabilistic inference to model human ability in event segmentation and generalization, the structured event memory (SEM) model successfully produces human-like event segmentation and identifies event schemata in naturalistic video data (Franklin et al, 2019).…”

What can narratives tell us about the neural bases of human memory?

“…Computational and theoretical modeling may address the mechanistic underpinnings of narrative memory, pushing the field beyond describing observed phenomena (e.g. [69]). Future studies may go beyond using narratives as encoding material to examine narrativization during retrieval, potentially relating memory to spontaneous and creative thinking [70].…”