Semantics-weighted lexical surprisal modeling of naturalistic functional MRI time-series during spoken narrative listening

Russo, Andrea G.; Martino, Maria De; Mancuso, Azzurra; Iaconetta, Giorgio; Manara, Renzo; Elia, Annibale; Laudanna, Alessandro; Salle, Francesco Di; Esposito, Fabrizio

doi:10.1016/j.neuroimage.2020.117281

Cited by 21 publications

(25 citation statements)

References 75 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, we controlled semantic similarity for each version of each text per condition following a probabilistic approach (Russo et al, 2020). Drawing on the mean frequency of use of content words, we calculated a semantic similarity index for each text per condition, formalized as the probability of encountering the target word given a set of content words, that is, P(P7 IF P1 ∩ P2 ∩ P3 ∩ P4 ∩ P5 ∩ P6 ∩ P8), where P1 to P6 correspond to the frequency distribution of context content words, P7 to the frequency distribution of the high predictable target word, and P8 to the frequency distribution of the low predictable word (see Supplementary Material).…”

Section: Lamentablemente éL Había Sido El Vencedormentioning

confidence: 99%

Modulating effects of adverb polarity on lexical-semantic prediction during interclause processing in Spanish

Cárdenas-Mancilla¹,

Cárdenas-Claros²,

Ferreira³

2022

Preprint

View full text Add to dashboard Cite

In the present study, we investigated the facilitation effects of lexical-semantic preactivation during interclause processing, in a sample of 22 Spanish L1 undergraduate students. Previous research has mainly examined nouns’ predictability effects at clause level and focused on semantic factors that precede text interpretation. Yet, it remains unclear if such effects occur when text interpretation is not only preceded but also fed into, for instance, by subjective clause adverbs expressing positive/negative polarity in Spanish. Drawing on a self-paced reading task, we measured response times at the target nouns and fitted mixed-effects models on the outcome variable to the targets. We observed longer response times at low predictable nouns when negative adverb polarity mediated interclause processing. This finding suggests that predictability effects may be detected at early processing cycles, even before the noun introducing the lexical-semantic update is perceived as current input, thus facilitating the construction of a coherent text model.

show abstract

Section: Lamentablemente éL Había Sido El Vencedormentioning

confidence: 99%

Modulating effects of adverb polarity on lexical-semantic prediction during interclause processing in Spanish

Cárdenas-Mancilla¹,

Cárdenas-Claros²,

Ferreira³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…These models are often rooted in computational algorithms that assign surprisal (the degree to which the word is expected given the preceding context) and entropy (the degree to which the word constrains upcoming linguistic content) values to each word in a sentence, reflecting how easily a word can be integrated given the left context and the overall statistics of the language (Futrell et al, 2020). Neuroimaging evidence indicates that activation in language-related brain areas correlates with difficulty as reflected by these information-theoretic measures (e.g., Russo et al, 2020;Henderson, Choi, Lowder, & Ferreira, 2016). Recent neural models of language processing locate the operation of combining two elements into a syntactic representation in Brodmann's area 44 of Broca's area, which forms a network for processing of syntactic complexity in combination with the superior temporal gyrus (Fedorenko & Blank, 2020;Zaccarella, Schell, & Friederici, 2017; for a competing view, see Matchin & Hickok, 2020).…”

Section: Psycholinguistic Issues In Sentence Processing Researchmentioning

confidence: 99%

Cortical Tracking of Speech: Toward Collaboration between the Fields of Signal and Sentence Processing

Beier

Chantavarin

Rehrig

et al. 2021

Journal of Cognitive Neuroscience

View full text Add to dashboard Cite

In recent years, a growing number of studies have used cortical tracking methods to investigate auditory language processing. Although most studies that employ cortical tracking stem from the field of auditory signal processing, this approach should also be of interest to psycholinguistics—particularly the subfield of sentence processing—given its potential to provide insight into dynamic language comprehension processes. However, there has been limited collaboration between these fields, which we suggest is partly because of differences in theoretical background and methodological constraints, some mutually exclusive. In this article, we first review the theories and methodological constraints that have historically been prioritized in each field and provide concrete examples of how some of these constraints may be reconciled. We then elaborate on how further collaboration between the two fields could be mutually beneficial. Specifically, we argue that the use of cortical tracking methods may help resolve long-standing debates in the field of sentence processing that commonly used behavioral and neural measures (e.g., ERPs) have failed to adjudicate. Similarly, signal processing researchers who use cortical tracking may be able to reduce noise in the neural data and broaden the impact of their results by controlling for linguistic features of their stimuli and by using simple comprehension tasks. Overall, we argue that a balance between the methodological constraints of the two fields will lead to an overall improved understanding of language processing as well as greater clarity on what mechanisms cortical tracking of speech reflects. Increased collaboration will help resolve debates in both fields and will lead to new and exciting avenues for research.

show abstract

“…The prediction of the next word from a given sequence also occurs in the human brain when continuously engaged in the generation of meaningful linguistic structures from the auditory streams of words perceived up to that moment, such as during natural listening 6 , 11 , 12 . In addition, it is an established principle of sensory perception that an attention mechanism is also needed in the human brain to improve the prediction mechanism via synergistic modulation of input signals 13 , 14 .…”

Section: Introductionmentioning

confidence: 99%

Explaining neural activity in human listeners with deep learning via natural language processing of narrative text

Russo

Ciarlo

Ponticorvo

et al. 2022

Sci Rep

View full text Add to dashboard Cite

Deep learning (DL) approaches may also inform the analysis of human brain activity. Here, a state-of-art DL tool for natural language processing, the Generative Pre-trained Transformer version 2 (GPT-2), is shown to generate meaningful neural encodings in functional MRI during narrative listening. Linguistic features of word unpredictability (surprisal) and contextual importance (saliency) were derived from the GPT-2 applied to the text of a 12-min narrative. Segments of variable duration (from 15 to 90 s) defined the context for the next word, resulting in different sets of neural predictors for functional MRI signals recorded in 27 healthy listeners of the narrative. GPT-2 surprisal, estimating word prediction errors from the artificial network, significantly explained the neural data in superior and middle temporal gyri (bilaterally), in anterior and posterior cingulate cortices, and in the left prefrontal cortex. GPT-2 saliency, weighing the importance of context words, significantly explained the neural data for longer segments in left superior and middle temporal gyri. These results add novel support to the use of DL tools in the search for neural encodings in functional MRI. A DL language model like the GPT-2 may feature useful data about neural processes subserving language comprehension in humans, including next-word context-related prediction.

show abstract

Semantics-weighted lexical surprisal modeling of naturalistic functional MRI time-series during spoken narrative listening

Cited by 21 publications

References 75 publications

Modulating effects of adverb polarity on lexical-semantic prediction during interclause processing in Spanish

Modulating effects of adverb polarity on lexical-semantic prediction during interclause processing in Spanish

Cortical Tracking of Speech: Toward Collaboration between the Fields of Signal and Sentence Processing

Explaining neural activity in human listeners with deep learning via natural language processing of narrative text

Contact Info

Product

Resources

About