Semantic representations during language comprehension are affected by context

Abstract

The meaning of words in natural language depends crucially on context. However, most neuroimaging studies of word meaning use isolated words and isolated sentences with little context. Because the brain may process natural language differently from how it processes simplified stimuli, there is a pressing need to determine whether prior results on word meaning generalize to natural language. We investigated this issue by directly comparing the brain representation of semantic information across four conditions that vary in context. fMRI was used to record human brain activity while four subjects (two female) read words presented in four different conditions: narratives (Narratives), isolated sentences (Sentences), blocks of semantically similar words (Semantic Blocks), and isolated words (Single Words). Using a voxelwise encoding model approach, we find two clear and consistent effects of increasing context. First, stimuli with more context (Narratives, Sentences) evoke brain responses with substantially higher SNR across bilateral visual, temporal, parietal, and prefrontal cortices compared to stimuli with little context (Semantic Blocks, Single Words). Second, increasing context increases the representation of semantic information across bilateral temporal, parietal, and prefrontal cortices at the group level. However, in individual subjects, only natural language stimuli (Narratives) consistently evoke widespread representation of semantic information across the cortical surface. These results show that context has large effects on both the quality of neuroimaging data and on the representation of meaning in the brain, and they imply that the results of neuroimaging studies that use stimuli with little context may not generalize well to the natural regime.Significance StatementContext is an important part of understanding the meaning of natural language, but most neuroimaging studies of meaning use isolated words and isolated sentences with little context. Here we examined whether the results of neuroimaging studies that use out-of-context stimuli generalize to natural language. We find that increasing context improves the quality of neuroimaging data and changes where semantic information is represented in the brain. These results suggest that findings from studies using out-of-context stimuli may not generalize to natural language used in daily life.

Computational Language Modeling and the Promise of In Silico Experimentation

“…While we are fully capable of identifying action words in isolation, it is not necessary that the brain employs the same networks to understand such words in real-world settings ( Matusz et al, 2019 ), for example, as used in a conversation or a story. In contrast to such studies, those using naturalistic stimuli have found more engagement and activation in higher order cortical regions, likely due to the incorporation of long-range structure ( Deniz et al, 2021 ; Lerner et al, 2011 ). Furthermore, due to practical limitations, controlled studies typically use small stimulus sets that span a limited domain.…”

The Cortical Representation of Language Timescales is Shared between Reading and Listening

“…This discrepancy perhaps explains why in Oota et al (2022), language models trained on higher-level tasks (e.g., summarization, paraphrase detection) were better able to predict listening than reading data. Our study used matched stimuli for reading and listening experiments, and the similarities we observed highlight the importance of using narrativelength, naturalistic stimuli to elicit brain representations of high-level linguistic features (Deniz et al, 2023).…”

“…Some methods require the use of stimuli that are scrambled at different temporal granularities (Lerner et al, 2011; Blank and Fedorenko, 2020). However, artificially scrambled stimuli may cause attentional shifts that evoke brain responses that are not representative of brain responses to natural stimuli (Hamilton and Huth, 2020; Hasson et al, 2010; Deniz et al, 2021). By using voxelwise modeling, we can instead estimate timescale selectivity with brain responses to ecologically valid stimuli.…”

“…Early methods required the use of stimuli that are scrambled at different temporal granularities (Lerner et al, 2011;Blank and Fedorenko, 2020). However, artificially scrambled stimuli may cause attentional shifts, evoking brain responses that are not representative of brain responses to natural stimuli (Hamilton and Huth, 2020;Hasson et al, 2010;Deniz et al, 2023). Other approaches measured the rate of change in patterns of brain responses in order to determine the temporal granularity of representations in each brain region (Baldassano et al, 2017).…”

Neural Dynamics of the Processing of Speech Features: Evidence for a Progression of Features from Acoustic to Sentential Processing

“…Previous research has shown numerous brain regions that are sensitive to specific aspects of language understanding [4][5][6] . However, focus on speech-based language understanding has been limited, compared to text-based language understanding.…”