Decoding the neural representation of story meanings across languages

Abstract

Drawing from a common lexicon of semantic units, humans fashion narratives whose meaning transcends that of their individual utterances. However, while brain regions that represent lower-level semantic units, such as words and sentences, have been identified, questions remain about the neural representation of narrative comprehension, which involves inferring cumulative meaning. To address these questions, we exposed English, Mandarin and Farsi native speakers to native language translations of the same stories during fMRI scanning. Using a new technique in natural language processing, we calculated the distributed representations of these stories (capturing the meaning of the stories in high-dimensional semantic space), and demonstrate that using these representations we can identify the specific story a participant was reading from the neural data. Notably, this was possible even when the distributed representations were calculated using stories in a different language than the participant was reading.Relying on over 44 billion classifications, our results reveal that identification relied on a collection of brain regions most prominently located in the default mode network.These results demonstrate that neuro-semantic encoding of narratives happens at levels higher than individual semantic units and that this encoding is systematic across both individuals and languages. NEURO-SEMANTIC REPRESENTATION OF STORIES 3 Decoding the Neural Representation of Story Meanings across LanguagesOne of the defining characteristics of human language is its capacity for semantic extensibility. Drawing from a common lexicon of morphemes and words, humans generate and comprehend sophisticated, higher-level utterances that transcend the sum of their individual units. This is perhaps best exemplified in stories, in which sequences of events invite inferences about the intentions and motivations of characters, about cause and effect, and about theme and message. The kind of meaning that emerges over time as one listens to a story is not easily captured by analysis at the word level alone.Further, a necessary condition for generating higher-level semantic constructs is that speakers of the same language infer similar meanings from expressions of both lower and higher level semantic units. For example, it can be assumed that when speakers of the same language listen to stories, the perceived meanings of these stories have much in NEURO-SEMANTIC REPRESENTATION OF STORIES 4In this work, our aim is to move beyond word-level semantics to investigate neuro-semantic representations at the story-level across three different languages.Specifically, we set out to determine if there are systematic patterns in the neuro-semantic representations of stories beyond those corresponding to word-level stimuli. Our aim is motivated by the long-standing understanding that discourse representations are different from the sum of all of their lexical or clausal parts. Most psycholinguistic models of discourse processing are concerned with the con...

Brain mechanisms underlying cue-based memorizing during free viewing of movie Memento

“…On the other hand, majority of human memory studies that use non-naturalistic stimuli have reported dominance of the left hemisphere (Johnson et al, 2013;Kim, 2011;Scalici et al, 2017), while for studies using longer-duration, naturalistic stimuli results are bilateral or right dominant (Chen et al, 2017;J. Chen et al, 2015a;Dehghani et al, 2017;Hasson et al, 2004;J€ a€ askel€ ainen et al, 2008;Kauppi et al, 2010;Lerner et al, 2011). Based on the hemispheric encoding/retrieval asymmetry model and supported by various neuroimaging studies, right PFC is typically engaged more than left in memory retrieval processes (Habib et al, 2003;Tulving et al, 1994).…”

“…A common factor for fMRI studies applying long-duration naturalistic stimulus appear to be involvement of DMN, particularly precuneus and ANG (Chen et al, 2017;J. Chen et al, 2015a;Dehghani et al, 2017;Hasson et al, 2004;J€ a€ askel€ ainen et al, 2008;Kauppi et al, 2010;Lerner et al, 2011). Precuneus is known as a functional core of the DMN (Utevsky et al, 2014).…”

“…We applied event-specific pattern analysis and hypothesized that our analyses implicate brain structures that have been previously associated with long-duration memory storing and narrative comprehension. Such regions are medial temporal lobe, frontal and prefrontal cortices, hippocampus, precuneus, angular gyrus, cingulate, middle temporal pole and frontal gyri (Bird et al, 2015;Chen et al, 2017;Dehghani et al, 2017;Kauttonen et al, 2015;Nadel and Hardt, 2011;Oedekoven et al, 2017;Summerfield et al, 2006;Wheeler et al, 1995;Yaffe et al, 2014). In addition, we assumed that the right hemisphere would dominate the cognitive processes related to long-duration narrative comprehension, based on previous findings (AbdulSabur et al, 2014;J€ a€ askel€ ainen et al, 2008;Marini et al, 2005;Tyl en et al, 2015;Xu et al, 2005).…”

Groups and Emotional Arousal Mediate Neural Synchrony and Perceived Ritual Efficacy

“…Additional studies have shown that face-to-face interaction and communication not only generates synchrony between two people but also predicts mutual prosociality (Jiang et al, 2012 , 2015 ; Hu et al, 2017 ). Both successful communication and memory encoding of narrative content in story segments have also been shown to produce synchrony between participants (Hasson et al, 2008 ; Dehghani et al, 2017 ). Future research on direct communication and common memory could elucidate differences between co-presence and symbolic interactionism.…”

Decoding the meaning of unconsciously processed words using fMRI-based MVPA

“…This is the first time that MVPA-based across-language generalization has been used to investigate the scope of non-conscious semantic representations. However, the same approach has already been used with positive results in a number of different fMRI studies where words were available to conscious awareness [Buchweitz et al (2012), Correia et al (2014), Zinszer et al (2015), Dehghani et al (2017)]. The factors leading to acrosslanguage generalization are not well understood.…”