Combining computational controls with natural text reveals new aspects of meaning composition

Abstract

To study a core component of human intelligence—our ability to combine the meaning of words—neuroscientists look for neural correlates of meaning composition, such as brain activity proportional to the difficulty of understanding a sentence. However, little is known about the product of meaning composition in the brain—the combined meaning of words beyond their individual meaning. We term this product “supra-word meaning” and devise a computational representation for it by using recent neural network algorithms and a new technique to disentangle composed-from individual-word meaning. Using functional magnetic resonance imaging, we find that supra-word meaning is processed in the bilateral anterior and posterior temporal lobes. Surprisingly, we cannot detect supra-word meaning in magnetoencephalography. These results suggest that composed meaning is maintained through a different neural mechanism that does not consist of synchronized cell firing. This difference in sensitivity has implications for past neuroimaging results and future neuroimaging studies.

Using surprisal and fMRI to map the neural bases of broad and local contextual prediction during natural language comprehension

“…In addition, we explored the neuro-anatomical More generally, this paper adds another data point demonstrating the relevance of tools from computational linguistics in cognitive neuroscience research Jain and Huth, 2018;Toneva et al, 2020) and the value of naturalistic stimuli in contextually situated and ecologically valid research (Maguire, 2012;Brennan, 2016;Hamilton and Huth, 2020).…”

“…1 As one salient example, Wehbe et al (2014) investigated how well vector representations predicted brain activity for subjects reading fiction, in their case material from Harry Potter and the Sorcerer's Stone, based on within-sentence context. Also working within the sentence using naturalistic listening, Toneva et al (2020) derived composed representations of "supraword meaning" using contextualized word representations (ELMo, Peters et al, 2018) to capture the compositional meaning of multi-word expressions and event/argument structure. Jain and Huth (2018) make predictions of neural activity using LSTM representations from up to the previous 20 words of context (which would be on the order of 8-10 seconds of speech on average).…”

Cortical processing of reference in language revealed by computational models

“…Anderson et al, n.d. ; Toneva et al, 2021; Schrimpf et al, 2021). However, many of the deep learning models are not intended to match the human cognitive process.…”

“…; J. Brennan et al, 2012, 2016; Jackson et al, 2021; Hale et al, 2018; Martin & Doumas, 2019; Toneva & Wehbe, 2019; Toneva et al, 2021; Schrimpf et al, 2021; Wehbe et al, 2014). Here we compared the fit of five computational models for pronoun processing with the fMRI and MEG data within the LMTG functional regions of interest (fROIs) using representational similarity analysis (Kriegeskorte et al, 2008).…”

“…Three of the models are knowledge-based symbolic models that formalize the Binding Theory (Hobbs, 1977), the Centering Theory (S. Brennan et al, 1987) and the memory account for pronoun resolution (van Rij et al, 2013), while the other two models are data-driven deep neural networks with either a long short-term memory (LSTM) architecture (ELMo; Lee et al, 2017; Lee et al,2018) or a Transformer architecture (BERT; Joshi et al, 2019). Both ELMo and BERT have shown high performance in Natural Language Processing and hold great promise for simulating the brain’s language system (Toneva et al, 2021; Schrimpf et al, 2021). However, our correlational results showed that the memory-based ACT-R model best explains the neural signatures for both the English and Chinese fMRI data and the MEG data, suggesting a domain-general mechanism for pronoun resolution that resembles memory retrieval.…”