Amplification of local changes along the timescale processing hierarchy

Abstract

Small changes in word choice can lead to dramatically different interpretations of narratives. How does the brain accumulate and integrate such local changes to construct unique neural representations for different stories? In this study, we created two distinct narratives by changing only a few words in each sentence (e.g., "he" to "she" or "sobbing" to "laughing") while preserving the grammatical structure across stories. We then measured changes in neural responses between the two stories. We found that differences in neural responses between the two stories gradually increased along the hierarchy of processing timescales. For areas with short integration windows, such as early auditory cortex, the differences in neural responses between the two stories were relatively small. In contrast, in areas with the longest integration windows at the top of the hierarchy, such as the precuneus, temporal parietal junction, and medial frontal cortices, there were large differences in neural responses between stories. Furthermore, this gradual increase in neural differences between the stories was highly correlated with an area's ability to integrate information over time. Amplification of neural differences did not occur when changes in words did not alter the interpretation of the story (e.g., sobbing to "crying"). Our results demonstrate how subtle differences in words are gradually accumulated and amplified along the cortical hierarchy as the brain constructs a narrative over time.timescale | amplification | fMRI | narrative | hierarchy S tories unfold over many minutes and are organized into temporarily nested structures: Paragraphs are made of sentences, which are made of words, which are made of phonemes. Understanding a story therefore requires processing the story at multiple timescales, starting with the integration of phonemes to words within a relatively short temporal window, to the integration of words to sentences across a longer temporal window of a few seconds, and up to the integration of sentences and paragraphs into a coherent narrative over many minutes. It was recently suggested that the temporally nested structure of language is processed hierarchically along the cortical surface (1-4). A consequence of the hierarchical structure of language is that small changes in word choice can give rise to large differences in sentence and overall narrative interpretation. Here, we propose that local momentary changes in linguistic input in the context of a narrative (e.g., "he" to "she" or "sobbing" to "laughing") are accumulated and amplified during the processing of linguistic content across the cortex. Specifically, we hypothesize that as areas in the brain increase in their ability to integrate information over time (e.g., processing "word" level content vs. "sentence" or "narrative" level content), the neural response to small word changes will become increasingly divergent.Previously, we defined a temporal receptive window (TRW) as the length of time during which prior information from an ongoing stimulu...

Shared neural and cognitive mechanisms in action and language: The multiscale information transfer framework

“…Furthermore, comprehension studies have provided evidence in support of a cortical processing Schematic representation illustrating the fact that identical probabilities of syntagmatic co-occurrence (e.g., p(VERB1 + OBJECT1) = 0.20 and p (VERB2 + OBJECT1) = 0.20) need not amount to equal expectedness, since paradigmatic competition between different options (here: objects) will also play a role. Thus, OBJECT1 is the paradigmatically dispreferred competitor after VERB1, while it is the preferred option after VERB2 hierarchy that is consistent with the functional scales of the MSIT framework (Grafton, 2009;de Heer, Huth, Griffiths, Gallant, & Theunissen, 2017;Yeshurun, Nguyen, & Hasson, 2017).…”

“…Consistent with this claim, recent language and action processing studies have revealed extensive neural activation overlap between the production and comprehension processes within each domain, respectively (Grafton, ; Silbert, Honey, Simony, Poeppel, & Hasson, ). Furthermore, comprehension studies have provided evidence in support of a cortical processing hierarchy that is consistent with the functional scales of the MSIT framework (Grafton, ; de Heer, Huth, Griffiths, Gallant, & Theunissen, ; Yeshurun, Nguyen, & Hasson, ).…”

Constructing and Forgetting Temporal Context in the Human Cerebral Cortex

“…We developed a set of models to explain prior measurements of neural responses to intact and temporally scrambled stimuli (e.g. Lerner et al, 2011, also Baldassano et al, 2017Chen et al, 2017;Farbood et al, 2015;Hasson et al, 2008;Honey et al, 2012;Simony et al, 2016;Yeshurun et al, 2017). In particular, lower-level sensory regions should display contextinvariant responses to the intact and scrambled narratives: the response to a particular input segment (e.g.…”

No evidence for functional distinctions across fronto-temporal language regions in their temporal receptive windows

“…Further posterior and anterior temporal regions could only track lists of sentences or paragraphs (but not words), indicative of sensitivity to phraseor sentence-level information. And, finally, some inferior frontal regions exhibited this same pattern of sensitivity to phrase/sentence information, with yet others reliably tracking only paragraph (but not sentence) lists, indicative of sensitivity to information above the sentence level (a long TRW).This hierarchy of integration timescales is an appealing organizing principle of the core language network (DeWitt and Rauschecker, 2012;Bornkessel-Schlesewsky et al, 2015;Hasson et al, 2015;Chen et al, 2016;Baldassano et al, 2017;Yeshurun et al, 2017a;Sheng et al, 2018). Nevertheless, there are several reasons to question the putative correspondence between this hierarchy and the set of language-selective cortical regions.…”

“…This hierarchy of integration timescales is an appealing organizing principle of the core language network (DeWitt and Rauschecker, 2012;Bornkessel-Schlesewsky et al, 2015;Hasson et al, 2015;Chen et al, 2016;Baldassano et al, 2017;Yeshurun et al, 2017a;Sheng et al, 2018). Nevertheless, there are several reasons to question the putative correspondence between this hierarchy and the set of language-selective cortical regions.…”