Audio-visual integration in noise: Influence of auditory and visual stimulus degradation on eye movements and perception of the McGurk effect

Stacey, Jemaine E.; Howard, Christina J.; Mitra, Suvobrata; Stacey, Paula C.

doi:10.3758/s13414-020-02042-x

Cited by 17 publications

(14 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We know from previous studies that visual context is more beneficial as acoustic degradation or noisy background increases [1][2][3]30], consistent with the principle of inverse effectiveness in multisensory integration [31,32]; but see [33]. Increased adversity in the acoustical environment may cause individuals to concentrate more on mouth movements [34,35], leading to greater engagement of the visual modality with the auditory modality. By using 24-channels vocoding, we were hoping to balance listening effort between the natural and degraded speech tokens, to rule out listening effort as a confound, while at the same time still examining neurophysiological sensitivity to semantic prediction.…”

Section: Discussionsupporting

confidence: 60%

The Cross-Modal Suppressive Role of Visual Context on Speech Intelligibility: An ERP Study

et al. 2020

View full text Add to dashboard Cite

The efficacy of audiovisual (AV) integration is reflected in the degree of cross-modal suppression of the auditory event-related potentials (ERPs, P1-N1-P2), while stronger semantic encoding is reflected in enhanced late ERP negativities (e.g., N450). We hypothesized that increasing visual stimulus reliability should lead to more robust AV-integration and enhanced semantic prediction, reflected in suppression of auditory ERPs and enhanced N450, respectively. EEG was acquired while individuals watched and listened to clear and blurred videos of a speaker uttering intact or highly-intelligible degraded (vocoded) words and made binary judgments about word meaning (animate or inanimate). We found that intact speech evoked larger negativity between 280–527-ms than vocoded speech, suggestive of more robust semantic prediction for the intact signal. For visual reliability, we found that greater cross-modal ERP suppression occurred for clear than blurred videos prior to sound onset and for the P2 ERP. Additionally, the later semantic-related negativity tended to be larger for clear than blurred videos. These results suggest that the cross-modal effect is largely confined to suppression of early auditory networks with weak effect on networks associated with semantic prediction. However, the semantic-related visual effect on the late negativity may have been tempered by the vocoded signal’s high-reliability.

show abstract

Section: Discussionsupporting

confidence: 60%

The Cross-Modal Suppressive Role of Visual Context on Speech Intelligibility: An ERP Study

et al. 2020

View full text Add to dashboard Cite

show abstract

“…In sum, previous research by Alsius et al ( 2005 ), Gurler et al ( 2015 ), Munhall et al ( 2009 ), and Stacey et al ( 2020 ) suggests that the degree to which people perceive the McGurk illusion depends (1) on their attention in general (with less attention leading to a decrease of the illusion) and (2) on their attentional focus on the speaker’s mouth versus elsewhere (with a focus on the mouth leading to an increase of the illusion).…”

Section: Introductionmentioning

confidence: 54%

Section: Multisensory Integration In Speech Perceptionmentioning

confidence: 82%

“…The results showed that the distribution of participants’ eye fixations predicted the degree to which participants perceived the McGurk illusion. Specifically, the more participants tended to look at the speaker’s mouth, the more they perceived the McGurk illusion—presumably because they were more strongly influenced by the visual signal provided by the lip movements (for a replication and additional manipulations, see Stacey et al, 2020 ). Hence, observing the speaker’s lip movements—a strategy that would usually improve speech comprehension (Ma et al, 2009 ; MacLeod, & Summerfield, 1987 ; Ross et al, 2007 )—has a negative and misleading effect because the visual signal coming from the lips is inconsistent with the auditory signal.…”

Section: Introductionmentioning

confidence: 99%

“…In particular, another’s direct eye gaze and motion onset (e.g., switching from direct to averted gaze, or from closed to open eyes) are two powerful cues that capture our attention (e.g., Abrams & Christ, 2003 ; Böckler et al, 2014 ; Senju & Hasegawa, 2005 ). In turn, what we attend (and to what extent we attend it) determines how we process incoming multisensory signals (e.g., Alsius et al, 2005 ; Munhall et al, 2009 ) – and thus affects whether we can be “tricked” by our senses and subjected to multisensory illusions (Gurler et al, 2015 ; Stacey et al, 2020 ).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

When eyes beat lips: speaker gaze affects audiovisual integration in the McGurk illusion

Wahn

Schmitz

Kingstone

et al. 2021

Psychological Research

View full text Add to dashboard Cite

Eye contact is a dynamic social signal that captures attention and plays a critical role in human communication. In particular, direct gaze often accompanies communicative acts in an ostensive function: a speaker directs her gaze towards the addressee to highlight the fact that this message is being intentionally communicated to her. The addressee, in turn, integrates the speaker’s auditory and visual speech signals (i.e., her vocal sounds and lip movements) into a unitary percept. It is an open question whether the speaker’s gaze affects how the addressee integrates the speaker’s multisensory speech signals. We investigated this question using the classic McGurk illusion, an illusory percept created by presenting mismatching auditory (vocal sounds) and visual information (speaker’s lip movements). Specifically, we manipulated whether the speaker (a) moved his eyelids up/down (i.e., open/closed his eyes) prior to speaking or did not show any eye motion, and (b) spoke with open or closed eyes. When the speaker’s eyes moved (i.e., opened or closed) before an utterance, and when the speaker spoke with closed eyes, the McGurk illusion was weakened (i.e., addressees reported significantly fewer illusory percepts). In line with previous research, this suggests that motion (opening or closing), as well as the closed state of the speaker’s eyes, captured addressees’ attention, thereby reducing the influence of the speaker’s lip movements on the addressees’ audiovisual integration process. Our findings reaffirm the power of speaker gaze to guide attention, showing that its dynamics can modulate low-level processes such as the integration of multisensory speech signals.

show abstract

Spatial and temporal (non)binding of audiovisual rhythms in sensorimotor synchronisation

et al. 2023

View full text Add to dashboard Cite

Human movement synchronisation with moving objects strongly relies on visual input. However, auditory information also plays an important role, since real environments are intrinsically multimodal. We used electroencephalography (EEG) frequency tagging to investigate the selective neural processing and integration of visual and auditory information during motor tracking and tested the effects of spatial and temporal congruency between audiovisual modalities. EEG was recorded while participants tracked with their index finger a red flickering (rate fV = 15 Hz) dot oscillating horizontally on a screen. The simultaneous auditory stimulus was modulated in pitch (rate fA = 32 Hz) and lateralised between left and right audio channels to induce perception of a periodic displacement of the sound source. Audiovisual congruency was manipulated in terms of space in Experiment 1 (no motion, same direction or opposite direction), and timing in Experiment 2 (no delay, medium delay or large delay). For both experiments, significant EEG responses were elicited at fV and fA tagging frequencies. It was also hypothesised that intermodulation products corresponding to the nonlinear integration of visual and auditory stimuli at frequencies fV ± fA would be elicited, due to audiovisual integration, especially in Congruent conditions. However, these components were not observed. Moreover, synchronisation and EEG results were not influenced by congruency manipulations, which invites further exploration of the conditions which may modulate audiovisual processing and the motor tracking of moving objects.

show abstract

Audio-visual integration in noise: Influence of auditory and visual stimulus degradation on eye movements and perception of the McGurk effect

Cited by 17 publications

References 43 publications

The Cross-Modal Suppressive Role of Visual Context on Speech Intelligibility: An ERP Study

The Cross-Modal Suppressive Role of Visual Context on Speech Intelligibility: An ERP Study

When eyes beat lips: speaker gaze affects audiovisual integration in the McGurk illusion

Spatial and temporal (non)binding of audiovisual rhythms in sensorimotor synchronisation

Contact Info

Product

Resources

About