Binding and unbinding the auditory and visual streams in the McGurk effect

Nahorna, Olha; Berthommier, Frédéric; Schwartz, Jean‐Luc

doi:10.1121/1.4728187

Cited by 65 publications

(103 citation statements)

References 80 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results are consistent with the previous findings (Nahorna et al 2012) Navarra et al 2010;Alsius et al 2014). Experiment B shows that attentional mechanisms may intervene at the level of single audiovisual sources in an audiovisual speech scene, selectively increasing or decreasing the amount of fusion depending on the coherence of the attended source.…”

Section: Response Timesupporting

confidence: 82%

Physiology, Psychoacoustics and Cognition in Normal and Impaired Hearing

Dijk¹,

Başkent²,

Gaudrain

et al. 2016

Advances in Experimental Medicine and Biology

View full text Add to dashboard Cite

Section: Response Timesupporting

confidence: 82%

Physiology, Psychoacoustics and Cognition in Normal and Impaired Hearing

Dijk¹,

Başkent²,

Gaudrain

et al. 2016

Advances in Experimental Medicine and Biology

View full text Add to dashboard Cite

“…However Exp. A. confirms that the contextual stimulus may modulate fusion as in our previous experiments (Nahorna et al 2012;Ganesh et al 2013;Nahorna et al 2015) and extends the concept to the case of competing sources.…”

Section: Discussionsupporting

confidence: 70%

“…In Exp. A, the participants were involved in a monitoring paradigm in which they were asked to constantly look at the screen and monitor for possible "ba" or "da" targets by pressing an appropriate key, as in Nahorna et al (2012Nahorna et al ( , 2015. In Exp.…”

Section: Methodsmentioning

confidence: 99%

“…To provide evidence for this "binding and fusion" AVSSA process, Nahorna et al (2012Nahorna et al ( , 2015 showed that the McGurk effect can be significantly and strongly reduced by an audiovisual context made of a few seconds of incoherent material (sounds and images coming from different speech sources) presented before the McGurk target (audio "ba" plus video "ga"): the target, classically perceived as "da" in the McGurk effect, was more often perceived as "ba", suggesting a decreased weight of the visual input in the fusion process. The interpretation was that the incoherent context resulted in an "unbinding" effect decreasing the visual weight and hence diminishing the McGurk effect.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Audio Visual Integration with Competing Sources in the Framework of Audio Visual Speech Scene Analysis

Ganesh

Berthommier

Schwartz

2016

Advances in Experimental Medicine and Biology

Self Cite

View full text Add to dashboard Cite

We introduce "Audio-Visual Speech Scene Analysis" (AVSSA) as an extension of the two-stage Auditory Scene Analysis model towards audiovisual scenes made of mixtures of speakers. AVSSA assumes that a coherence index between the auditory and the visual input is computed prior to audiovisual fusion, enabling to determine whether the sensory inputs should be bound together. Previous experiments on the modulation of the McGurk effect by audiovisual coherent vs. incoherent contexts presented before the McGurk target have provided experimental evidence supporting AVSSA. Indeed, incoherent contexts appear to decrease the McGurk effect, suggesting that they produce lower audiovisual coherence hence less audiovisual fusion. The present experiments extend the AVSSA paradigm by creating contexts made of competing audiovisual sources and measuring their effect on McGurk targets. The competing audiovisual sources have respectively a high and a low audiovisual coherence (that is, large vs. small audiovisual comodulations in time). The first experiment involves contexts made of two auditory sources and one video source associated to either the first or the second audio source. It appears that the McGurk effect is smaller after the context made of the visual source associated to the auditory source with less audiovisual coherence. In the second experiment with the same stimuli, the participants are asked to attend to either one or the other source. The data show that the modulation of fusion depends on the attentional focus. Altogether, these two experiments shed light on audiovisual binding, the AVSSA process and the role of attention. Keywords

show abstract

“…On the other hand, exposure to as few as five successive presentations of phonetically incoherent (e.g. "ra-ka") AV syllables presented before the McGurk ("da") syllable can substantially reduce the prevalence of the illusion (Nahorna et al 2012). Importantly, expectations can affect stimulus processing even when based on irrelevant stimuli: Lateralised targets are detected faster in unattended spatial locations if they appear in sync with a rhythmic irrelevant stimulus (Jones 2015).…”

Section: Stimulus Statistics and Beyondmentioning

confidence: 99%

The COGs (context, object, and goals) in multisensory processing

et al. 2016

View full text Add to dashboard Cite

Our understanding of how perception operates in real-world environments has been substantially advanced by studying both multisensoryprocesses and "top-down" control processes influencing sensory processing via activity from higher-order brain areas, such as attention, memory, and expectations. As the two topicshave been traditionally studied separately, the mechanisms orchestrating real-world multisensory processingremain unclear.Past work has revealed that the observer's goals gate the influence of many multisensory processes on brain and behavioural responses, whereas some other multisensory processes mightoccur independently of these goals. Consequently, other forms of top-down control beyond goaldependence are necessaryto explain the full range of multisensory effects currently reported at the brain and the cognitive level. These forms of control includesensitivity to stimulus context as well as the detection of matches(or lack thereof) between a multisensory stimulus and categorical attributes of naturalistic objects (e.g. tools, animals). In this review we discuss and integrate the existing findings that demonstrate the importance of such goal-, object-and context-based top-down control over multisensory processing. We then put forward a few principles emerging from this literature review with respect to the mechanisms underlying multisensory processing, and discuss their possible broader implications. Attention, multisensory, control, object, audiovisual, brain mapping 3 Research from the past 30 years has demonstrated a whole range of behavioural benefits engendered by integrating information across the senses (multisensory integration, MSI), including faster motor responses and facilitated object recognition in noisy environments (e.g. Stein 2012). In a separate and independent manner, studies employing unisensory stimuli have been critically advancing our understanding of the nature and the importance of top-down mechanisms that control information processing. Thetopdown nature of these mechanisms lies in that they shape perceptual processing of new inputs by activating information stored in higher-order brain areas (e.g. Summerfield and Egner 2009). Key words:Studies of top-down control have traditionally focused on attentional (i.e. goaldependent)mechanisms, which promote the processing of stimulior objects in the environment that areimportant tothe current behavioural goals of the observer (e.g.Desimone and Duncan 1995). These mechanisms enhance the processing of stimuli appearing in task-relevant spatial locations and/or moments in time. These mechanisms likewise facilitate the processingof those stimuli whose attributes (e.g. colour red), feature dimensions (e.g. shape) or identity (e.g. a particular face) match the observer's goals.Simultaneously, it has been increasingly recognised in the literature that information processing issensitive to other types oftop-down processes, principally those gauged by thememory of past stimulation and one's expectations (see Figure 1A for a summar...

show abstract

Binding and unbinding the auditory and visual streams in the McGurk effect

Cited by 65 publications

References 80 publications

Physiology, Psychoacoustics and Cognition in Normal and Impaired Hearing

Physiology, Psychoacoustics and Cognition in Normal and Impaired Hearing

Audio Visual Integration with Competing Sources in the Framework of Audio Visual Speech Scene Analysis

The COGs (context, object, and goals) in multisensory processing

Contact Info

Product

Resources

About