Audio-visual speech scene analysis: Characterization of the dynamics of unbinding and rebinding the McGurk effect

Abstract

While audiovisual interactions in speech perception have long been considered as automatic, recent data suggest that this is not the case. In a previous study, Nahorna et al. [(2012). J. Acoust. Soc. Am. 132, 1061-1077] showed that the McGurk effect is reduced by a previous incoherent audiovisual context. This was interpreted as showing the existence of an audiovisual binding stage controlling the fusion process. Incoherence would produce unbinding and decrease the weight of the visual input in fusion. The present paper explores the audiovisual binding system to characterize its dynamics. A first experiment assesses the dynamics of unbinding, and shows that it is rapid: An incoherent context less than 0.5 s long (typically one syllable) suffices to produce a maximal reduction in the McGurk effect. A second experiment tests the rebinding process, by presenting a short period of either coherent material or silence after the incoherent unbinding context. Coherence provides rebinding, with a recovery of the McGurk effect, while silence provides no rebinding and hence freezes the unbinding process. These experiments are interpreted in the framework of an audiovisual speech scene analysis process assessing the perceptual organization of an audiovisual speech input before decision takes place at a higher processing stage.

Audio Visual Integration with Competing Sources in the Framework of Audio Visual Speech Scene Analysis

“…However Exp. A. confirms that the contextual stimulus may modulate fusion as in our previous experiments (Nahorna et al 2012;Ganesh et al 2013;Nahorna et al 2015) and extends the concept to the case of competing sources.…”

“…The stimuli were similar to those of the previous experiment by (Nahorna et al 2015) with suitable modification in the paradigm. They were prepared from two sets of audiovisual material, a "syllables" material and a "sentences" material, produced by a French male speaker, with lips painted in blue to allow precise video analysis of lip movements (Lallouache 1990).…”

“…In Exp. A, the participants were involved in a monitoring paradigm in which they were asked to constantly look at the screen and monitor for possible "ba" or "da" targets by pressing an appropriate key, as in Nahorna et al (2012Nahorna et al ( , 2015. In Exp.…”

“…It consisted of a sequence of 2 or 4 syllables (A-syl-2 or A-syl-4) randomly extracted from the set "pa," "ta," "va," "fa," "za," "sa," "ka," "ra," "la," "ja," "cha," "ma," or "na," mixed with a portion of a random set of sentences with the adequate duration (A-sent-2 or A-sent-4). The 2-vs. 4-syllable duration was selected from earlier experiments by Nahorna et al (2015), showing that the effect of incoherent context was maximal (maximal reduction of the McGurk effect) for short 2-syllable contexts and slightly less for longer 4-syllable contexts. The visual components of the context were the visual stream associated with either the auditory syllables (V-syl-2 or V-syl-4) or the auditory sentences (V-sent-2 or V-sent-4).…”

“…To provide evidence for this "binding and fusion" AVSSA process, Nahorna et al (2012Nahorna et al ( , 2015 showed that the McGurk effect can be significantly and strongly reduced by an audiovisual context made of a few seconds of incoherent material (sounds and images coming from different speech sources) presented before the McGurk target (audio "ba" plus video "ga"): the target, classically perceived as "da" in the McGurk effect, was more often perceived as "ba", suggesting a decreased weight of the visual input in the fusion process. The interpretation was that the incoherent context resulted in an "unbinding" effect decreasing the visual weight and hence diminishing the McGurk effect.…”