The contribution of audiovisual speech to lexical-semantic processing in natural spoken sentences

Abstract

In everyday communication, natural spoken sentences are expressed in a multisensory way through auditory signals and speakers' visible articulatory gestures. An important issue is to know whether audiovisual speech plays a main role in the linguistic encoding of an utterance until access to meaning. To this end, we conducted an event-related potential experiment during which participants listened passively to spoken sentences and a lexical recognition task. The results revealed that N200 and N400 waves had a greater amplitude after semantically incongruous words than after expected words. This effect of semantic congruency was increased over N200 in the audiovisual trials. Words presented audiovisually also elicited a reduced amplitude of the N400 wave and a facilitated recovery in memory. Our findings shed light on the influence of audiovisual speech on the understanding of natural spoken sentences by acting on the early stages of word recognition in order to access a lexical-semantic network.

Oscillatory correlates of linguistic prediction and modality effects during listening to auditory-only and audiovisual sentences

“…Surprisingly, low gamma activity was strongly decreased by the audiovisual modality in comparison with the auditory-only modality during adjective recognition from 0.25 s. This is again consistent with previous studies showing that audiovisual speech contributes to word recognition (e.g., Brunellière et al, 2013Brunellière et al, , 2020Buchwald et al, 2009;Fort et al, 2013). The occurrence of such adjectives in speech input can be predicted timely thanks to the parenthetical structure of sentence context in our study (for other EEG experiments with similar structures, see Brunellière et al, 2019;Brunellière et al, 2020). This can be more salient with audiovisual sentences (Brunellière et al, 2020).…”

“…In line with previous studies showing an impact of audiovisual speech during the processing of linguistic information (e.g., Brunellière et al, 2013Brunellière et al, , 2020Buchwald et al, 2009;Fort et al, 2013), audiovisual speech benefits were observed during listening to sentence contexts and target words embedded in sentential context at various frequency bands. Alpha and beta oscillations were found to decrease when semantically constraining sentences were delivered in the audiovisual modality, compared to the auditory-only modality.…”

“…The occurrence of such adjectives in speech input can be predicted timely thanks to the parenthetical structure of sentence context in our study (for other EEG experiments with similar structures, see Brunellière et al, 2019;Brunellière et al, 2020). This can be more salient with audiovisual sentences (Brunellière et al, 2020). Visual information thus increases sensitivity to expected sensory information by temporal predictions (Peelle, & Sommers, 2015, for a review), which is in line with the stronger attentional engagement directed towards the adjective observed in alpha activity from listening to audiovisual sentences.…”

“…Participants listened to 16 practice sentences prior to the set of four 10-min blocks of 60 trials, each containing sentences from all experimental conditions and fillers presented in random order. During the experiment, they were instructed to listen to the sentences carefully for comprehension, without any further tasks (for similar approaches, see Van Den Brink & Hagoort, 2004, Brunellière et al, 2020. After the listening task, participants performed a lexical recognition task to examine whether predictive mechanisms induced a memory trace of expected (although not presented) words (see Figure 1C).…”

“…This study aims at investigating this issue to obtain a realistic view of the dynamic brain processes involved in spoken language comprehension. Although audiovisual speech may offer substantial benefits of processing at different levels, including sub-lexical, lexical and sentence levels (e.g., Basirat et al, 2018;Brunellière et al, 2013Brunellière et al, , 2020, neuronal oscillations associated with linguistic prediction when listening to unimodal and bimodal sentences have not yet been investigated. Using the same approach as previous electrophysiological studies focusing on linguistic predictive processing of sentences (Foucart et al, 2015;Otten et al, 2007;Van Berkum et al, 2005;Wicha et al, 2003), we presented semantically constraining spoken sentences followed by a possessive adjective that either matched the gender of the expected (albeit not presented) word or not (see Figure 1).…”