Combined predictive effects of sentential and visual constraints in early audiovisual speech processing

Abstract

In language comprehension, a variety of contextual cues act in unison to render upcoming words more or less predictable. As a sentence unfolds, we use prior context ( sentential constraints ) to predict what the next words might be. Additionally, in a conversation, we can predict upcoming sounds through observing the mouth movements of a speaker ( visual constraints ). In electrophysiological studies, effects of visual constraints have typically been observed early in language processing, while effects of sentential constraints have typically been observed later. We hypothesized that the visual and the sentential constraints might feed into the same predictive process such that effects of sentential constraints might also be detectable early in language processing through modulations of the early effects of visual salience. We presented participants with audiovisual speech while recording their brain activity with magnetoencephalography. Participants saw videos of a person saying sentences where the last word was either sententially constrained or not, and began with a salient or non-salient mouth movement. We found that sentential constraints indeed exerted an early (N1) influence on language processing. Sentential modulations of the N1 visual predictability effect were visible in brain areas associated with semantic processing, and were differently expressed in the two hemispheres. In the left hemisphere, visual and sentential constraints jointly suppressed the auditory evoked field, while the right hemisphere was sensitive to visual constraints only in the absence of strong sentential constraints. These results suggest that sentential and visual constraints can jointly influence even very early stages of audiovisual speech comprehension.

Altered Processing of Social Emotions in Individuals With Autistic Traits

“…In agreement with prior studies (Solberg Økland et al, 2019;Hosaka et al, 2021), the N1 amplitudes elicited by the social audio recordings (human voices) in Experiment 2 were larger than those elicited by the non-social audio recordings (non-human sounds). Since the N1 component in the auditory modality is an index of early auditory processing (Zhao et al, 2016), and focused auditory attention could result in larger N1 amplitudes in response to audio recordings (Tumber et al, 2014), our findings suggest that social audio recordings capture more attention than non-social audio recordings.…”

Altered processing of social emotions in individuals with autistic traits

“…In agreement with prior studies [67,68], the N1 amplitudes elicited by the social audio recordings (human voices) in Experiment 2 were larger than those elicited by the nonsocial audio recordings (non-human sounds). Since the N1 component in the auditory modality is an index of early auditory processing [69], and focused auditory attention could result in larger N1 amplitudes in response to audio recordings [33], our findings suggest that social audio recordings capture more attention than nonsocial audio recordings.…”

The role of iconic gestures and mouth movements in face-to-face communication

“…In contrast to iconic gestures, mouth movements are primarily useful in decoding the phonological information and listeners benefit from audiovisual speech because facial gestures can support predictions for upcoming words (Solberg Økland et al, 2019). This has been captured by the notion of visemes-that is, the shape(s) of the lips that correspond to a particular phoneme or group of phonemes (Fisher, 1968;Massaro & Cohen, 1995).…”

“…The majority of previous studies have only looked at the impact of one visual cue: iconic gestures or mouth movements, while the other cue was eliminated to achieve control. Thus, the face is cropped or covered in studies of gestures (e.g., Drijvers & Özyürek, 2017;Habets et al, 2010;Hirata & Kelly, 2010;Holle & Gunter, 2007;Holle et al, 2010), and the hands are not visible in studies of audiovisual speech (e.g., Ross et al, 2007;Solberg Økland et al, 2019;Tye-Murray et al, 2007). Only a handful of studies have investigated both gestures and mouth movements (Drijvers & Özyürek, 2017(Drijvers & Özyürek, , 2020Drijvers et al, 2019;Hirata & Kelly, 2010;Skipper et al, 2009;Zhang, Ding, et al, 2021a;Zhang, Frassinelli, et al, 2021b).…”