Narrative Event Segmentation in the Cortical Reservoir

Abstract

During continuous perception of movies or stories, awake humans display cortical activity patterns that reveal hierarchical segmentation of event structure. Sensory areas like auditory cortex display high frequency segmentation related to the stimulus, while semantic areas like posterior middle cortex display a lower frequency segmentation related to transitions between events (Baldassano et al. 2017). These hierarchical levels of segmentation are associated with different time constants for processing. Chien and Honey (2020) observed that when two groups of participants heard the same sentence in a narrative, preceded by different contexts, neural responses for the groups were initially different and then gradually aligned. The time constant for alignment followed the segmentation hierarchy: sensory cortices aligned most quickly, followed by mid-level regions, while some higher-order cortical regions took more than 10 seconds to align. These hierarchical segmentation phenomena can be considered in the context of processing related to comprehension. Uchida et al. (2021) recently described a model of discourse comprehension where word meanings are modeled by a language model pre-trained on a billion word corpus (Yamada et al 2020). During discourse comprehension, word meanings are continuously integrated in a recurrent cortical network. The model demonstrates novel discourse and inference processing, in part because of two fundamental characteristics: real-world event semantics are represented in the word embeddings, and these are integrated in a reservoir network which has an inherent gradient of functional time constants due to the recurrent connections. Here we demonstrate how this model displays hierarchical narrative event segmentation properties. The reservoir produces activation patterns that are segmented by the HMM of Baldassano et al (2017) in a manner that is comparable to that of humans. Context construction displays a continuum of time constants across reservoir neuron subset, while context forgetting has a fixed time constant across these subsets. Virtual areas formed by subgroups of reservoir neurons with faster time constants segmented with shorter events, while those with longer time constants preferred longer events. This neurocomputational recurrent neural network simulates narrative event processing as revealed by the fMRI event segmentation algorithm of Baldassano et al (2017), and provides a novel explanation of the asymmetry in narrative forgetting and construction observed by Chien and Honey (2020). The model extends the characterization of online integration processes in discourse to more extended narrative, and demonstrates how reservoir computing provides a useful model of cortical processing of narrative structure.

Correspondence between the layered structure of deep language models and temporal structure of natural language processing in the human brain

“…Together, these results hint that a deep language model with stacked recurrent networks may better fit the human brain’s neural architecture for processing natural language. Interestingly, there have been several attempts to develop such new architectures, such as universal transformers (Dehghani et al, 2018; Lan et al, 2019) and reservoir computing (Dominey, 2021). Future studies will have to compare how the internal processing of natural language compares between these models and the brain.…”

Information flow across the cortical timescales hierarchy during narrative construction

“…The cortical hierarchy of TRWs in humans has been described using fMRI (Chien & Honey, 2020;Hasson et al, 2008;Lerner et al, 2011;Yeshurun et al, 2017) and ECoG (Honey et al, 2012). Recent work has shown that deep language models also learn a gradient or hierarchy of increasing TRWs (Dominey, 2021;Peters et al, 2018;Vig & Belinkov, 2019), and that manipulating the temporal coherence of narrative input to a deep language model yields representations closely matching the cortical hierarchy of TRWs in the human brain (Caucheteux et al, 2021). Furthermore, the cortical hierarchy of TRWs matches the intrinsic processing timescales observed during rest in humans (Honey et al, 2012;Raut et al, 2020;Stephens et al, 2013) and monkeys (Murray et al, 2014).…”

Dissociable Neural Mechanisms for Human Inference Processing Predicted by Static and Contextual Language Models

“…(Radvansky & Zacks, 2017)). Indeed, mounting neuroscientific evidence shows how event structure is processed in the brain in terms of the spatial and temporal distributions of narrative event structure (Baldassano et al, 2017;Baldassano, Hasson, & Norman, 2018), and recurrent models of cortical processing are beginning to provide neurocomputational explanations for these event driven phenomenon (Dominey, 2021). What remains to further explore is the link between the computational process such as those we explore here, and the spatiotemporal distribution of neural activity observed in the brain during narrative comprehension that relies on inference.…”