Behavioural evidence for separate mechanisms of audiovisual temporal binding as a function of leading sensory modality

Abstract

The ability to integrate auditory and visual information is critical for effective perception and interaction with the environment, and is thought to be abnormal in some clinical populations. Several studies have investigated the time window over which audiovisual events are integrated, also called the temporal binding window, and revealed asymmetries depending on the order of audiovisual input (i.e. the leading sense). When judging audiovisual simultaneity, the binding window appears narrower and non-malleable for auditory-leading stimulus pairs and wider and trainable for visual-leading pairs. Here we specifically examined the level of independence of binding mechanisms when auditory-before-visual vs. visual-before-auditory input is bound. Three groups of healthy participants practiced audiovisual simultaneity detection with feedback, selectively training on auditory-leading stimulus pairs (group 1), visual-leading stimulus pairs (group 2) or both (group 3). Subsequently, we tested for learning transfer (crossover) from trained stimulus pairs to non-trained pairs with opposite audiovisual input. Our data confirmed the known asymmetry in size and trainability for auditory-visual vs. visual-auditory binding windows. More importantly, practicing one type of audiovisual integration (e.g. auditory-visual) did not affect the other type (e.g. visual-auditory), even if trainable by within-condition practice. Together, these results provide crucial evidence that audiovisual temporal binding for auditory-leading vs. visual-leading stimulus pairs are independent, possibly tapping into different circuits for audiovisual integration due to engagement of different multisensory sampling mechanisms depending on leading sense. Our results have implications for informing the study of multisensory interactions in healthy participants and clinical populations with dysfunctional multisensory integration.

The phase of pre‐stimulus brain oscillations correlates with cross‐modal synchrony perception

“…The CIs revealed that the A50V point (73 ± 16 ms) was estimated with higher precision than the V50A point (82 ± 27 ms; t (26) = 2.14, P = 0.04), suggesting less reliability for time estimation of vision leading trials compared to audition leading trials (individual data are reported in Supplementary Table S1). This results are well in line with previous results showing that synchrony perception has greater tolerance and flexibility for vision leading than audition leading asynchronies (Cecere et al., ; Stevenson & Wallace, ; Van Eijk et al., ; Yarrow et al., ). This result can be considered as evidence for the reliability of each sensory modality on our simultaneity judgement task.…”

“…This larger perceptual variability for vision‐first events has been noted in many previous psychophysical studies, when asymmetries have been sought for (Yarrow et al., ), and is probably related to the well known lower sensitivity in asynchrony detection when vision leads audition (Cecere, Gross, Willis, & Thut, ; Conrey & Pisoni, ; Stevenson & Wallace, ; Van Wassenhove et al., ). A second, perhaps complementary, explanation for the asymmetry in our results, is that synchrony perception of different modality orders might rely on distinct, partially non‐overlapping (modality specific) neural processes (Cecere et al., , ; Thorne & Debener, ). This explanation might not seem parsimonious at first, but its plausibility is supported after recent findings of analogous asymmetries in synchrony processing across different modality orders in other electrophysiological studies (Kaganovich & Schumaker, ; Kösem et al., ).…”

“…This explanation might not seem parsimonious at first, but its plausibility is supported after recent findings of analogous asymmetries in synchrony processing across different modality orders in other electrophysiological studies (Kaganovich & Schumaker, ; Kösem et al., ). In addition, the finding that perceptual learning for vision leading pairs is greater than for audition leading events (Cecere et al., ; Powers, Hillock, & Wallace, ) has been interpreted according to separate processes governing synchrony perception as a function of the leading modality (Cecere et al., ). It is therefore not so implausible that cross‐modal perception of time might rely on some sensory‐specific processes, bound to express differently as a function of modality order.…”

“…The SOA sign denotes the modality order of presentation (negative = AV; positive = VA). Because of the well known asymmetry in synchrony perception (VA pairs need longer SOA compared to AV pairs see Cecere, Gross, & Thut, ; Stevenson & Wallace, ; Van Eijk, Kohlrausch, Juola, & Van de Par, and Yarrow, Jahn, Durant, & Arnold, ), we decided to use different SOA endpoints for VA pairs (+412 ms) and AV pairs (−353 ms). Each trial started with a white fixation cross at the centre on the place holder, which served as warning signal, followed by the cross‐modal stimulus pair after a variable interval randomly chosen between 1,047 and 2,100 ms.…”

Tolerance for audiovisual asynchrony is enhanced by the spectrotemporal fidelity of the speaker’s mouth movements and speech

“…As the speed of speech increases, the importance of visual precedence for intelligibility becomes even more pronounced (Brungart, Iyer, Simpson, & Wassenhove, 2008), suggesting that this asymmetry is tuned to allow maximal comprehension of rapid conversational speech. This asymmetrical resizing of AV integration window may proximally result from separate auditory and visual initiated processes (Munhall & Tohkura, 1998; Cecere, Gross, & Thut, 2016; van Wassenhove, 2013) – a view consistent with the Dynamic Reweighting Model.…”