More than Words: Neurophysiological Correlates of Semantic Dissimilarity Depend on Comprehension of the Speech Narrative

Abstract

Speech comprehension relies on the ability to understand the meaning of words within a coherent context. Recent studies have attempted to obtain electrophysiological indices of this process by modelling how brain activity is affected by a word’s semantic dissimilarity to preceding words. While the resulting indices appear robust and are strongly modulated by attention, it remains possible that, rather than capturing the contextual understanding of words, they may actually reflect word-to-word changes in semantic content without the need for a narrative-level understanding on the part of the listener. To test this possibility, we recorded EEG from subjects who listened to speech presented in either its original, narrative form, or after scrambling the word order by varying amounts. This manipulation affected the ability of subjects to comprehend the narrative content of the speech, but not the ability to recognize the individual words. Neural indices of semantic understanding and low-level acoustic processing were derived for each scrambling condition using the temporal response function (TRF) approach. Signatures of semantic processing were observed for conditions where speech was unscrambled or minimally scrambled and subjects were able to understand the speech. The same markers were absent for higher levels of scrambling when speech comprehension dropped below chance. In contrast, word recognition remained high and neural measures related to envelope tracking did not vary significantly across the different scrambling conditions. This supports the previous claim that electrophysiological indices based on the semantic dissimilarity of words to their context reflect a listener’s understanding of those words relative to that context. It also highlights the relative insensitivity of neural measures of low-level speech processing to speech comprehension.

Heard or understood? Neural tracking of language features in a comprehensible story, an incomprehensible story and a word list

“…Overall our results regarding the observed patterns in the TRFs to language features converge with the findings of Di Liberto et al (2021) and Broderick et al (2020): the worse the listener can integrate a word in its context due to lower language proficiency or higher word scrambling, the less prominent the N400 20 response. Therefore, activation of a word and integration of the word in its context are more difficult, explaining why no significant clusters are found for both word surprisal and word frequency for the Frisian story.…”

“…By comparing listeners listening to their second language to native speakers of the language, Di Liberto et al (2021) showed that as language proficiency increases, the response around 400 ms becomes more similar to a native speaker. Two other studies manipulated the amount of speech comprehension by manipulating the speech via word scrambling (Broderick et al, 2020) or manipulating the speech rate (Verschueren et al, 2022). Both studies observed that as the speech became incomprehensible, the N400 disappeared.…”

Deep language algorithms predict semantic comprehension from brain activity

“…Second, several quantitative approaches have been proposed to investigate comprehension, either with "model-free" methods based on inter-subject correlation (e.g. 33,35,45 ) or "model-based" methods based on word vectors 46 . For example, Lerner et al analyzed the fMRI activity of subjects listening to either normal texts or texts scrambled at the word, sentence or paragraph level 33 .…”

“…While brain activity correlated across subjects in the primary and secondary auditory areas even when the input was heavily scrambled (and thus poorly comprehensible), the bilateral infero-frontal and temporo-parietal cortex only correlated across subjects when sentences and/or paragraphs were not scrambled (and thus comprehensible). Broderick et al used a similar design to investigate electro-encephalography (EEG) responses to variably scrambled versions of the same story 46 , as well as the EEG responses to speech played in reverse and in noise 47 . Consistently with our results, they showed that the mapping between word embeddings' and the EEG activity varies with comprehension as manipulated by these various protocols.…”

Effects of Age on Cortical Tracking of Word-Level Features of Continuous Competing Speech

“…However, due to the exploratory nature of these analyses, and because similar trends were not observed for SSQ m , it remains unclear if this neuro-behavioral association is reliable. A follow-up study explicitly manipulating comprehension difficulty of speech materials while maintaining lower-level intelligibility (e.g., by presenting in-vs. out-of-order speech segments; see Broderick et al, 2020) may help to further explore this link.…”