Being First Matters: Topographical Representational Similarity Analysis of ERP Signals Reveals Separate Networks for Audiovisual Temporal Binding Depending on the Leading Sense

Abstract

In multisensory integration, processing in one sensory modality is enhanced by complementary information from other modalities. Intersensory timing is crucial in this process because only inputs reaching the brain within a restricted temporal window are perceptually bound. Previous research in the audiovisual field has investigated various features of the temporal binding window, revealing asymmetries in its size and plasticity depending on the leading input: auditory–visual (AV) or visual–auditory (VA). Here, we tested whether separate neuronal mechanisms underlie this AV–VA dichotomy in humans. We recorded high-density EEG while participants performed an audiovisual simultaneity judgment task including various AV–VA asynchronies and unisensory control conditions (visual-only, auditory-only) and tested whether AV and VA processing generate different patterns of brain activity. After isolating the multisensory components of AV–VA event-related potentials (ERPs) from the sum of their unisensory constituents, we ran a time-resolved topographical representational similarity analysis (tRSA) comparing the AV and VA ERP maps. Spatial cross-correlation matrices were built from real data to index the similarity between the AV and VA maps at each time point (500 ms window after stimulus) and then correlated with two alternative similarity model matrices: AVmaps = VAmaps versus AVmaps ≠ VAmaps. The tRSA results favored the AVmaps ≠ VAmaps model across all time points, suggesting that audiovisual temporal binding (indexed by synchrony perception) engages different neural pathways depending on the leading sense. The existence of such dual route supports recent theoretical accounts proposing that multiple binding mechanisms are implemented in the brain to accommodate different information parsing strategies in auditory and visual sensory systems.SIGNIFICANCE STATEMENT Intersensory timing is a crucial aspect of multisensory integration, determining whether and how inputs in one modality enhance stimulus processing in another modality. Our research demonstrates that evaluating synchrony of auditory-leading (AV) versus visual-leading (VA) audiovisual stimulus pairs is characterized by two distinct patterns of brain activity. This suggests that audiovisual integration is not a unitary process and that different binding mechanisms are recruited in the brain based on the leading sense. These mechanisms may be relevant for supporting different classes of multisensory operations, for example, auditory enhancement of visual attention (AV) and visual enhancement of auditory speech (VA).

The phase of pre‐stimulus brain oscillations correlates with cross‐modal synchrony perception

“…First, the lower reliability of behavioural measures for vision‐leading pairs (in our case, seen as a tendency for wider confidence intervals for perceived synchrony boundaries) might have blurred the detection of phase alignment in pre‐stimulus EEG. This larger perceptual variability for vision‐first events has been noted in many previous psychophysical studies, when asymmetries have been sought for (Yarrow et al., ), and is probably related to the well known lower sensitivity in asynchrony detection when vision leads audition (Cecere, Gross, Willis, & Thut, ; Conrey & Pisoni, ; Stevenson & Wallace, ; Van Wassenhove et al., ). A second, perhaps complementary, explanation for the asymmetry in our results, is that synchrony perception of different modality orders might rely on distinct, partially non‐overlapping (modality specific) neural processes (Cecere et al., , ; Thorne & Debener, ).…”

“…This larger perceptual variability for vision‐first events has been noted in many previous psychophysical studies, when asymmetries have been sought for (Yarrow et al., ), and is probably related to the well known lower sensitivity in asynchrony detection when vision leads audition (Cecere, Gross, Willis, & Thut, ; Conrey & Pisoni, ; Stevenson & Wallace, ; Van Wassenhove et al., ). A second, perhaps complementary, explanation for the asymmetry in our results, is that synchrony perception of different modality orders might rely on distinct, partially non‐overlapping (modality specific) neural processes (Cecere et al., , ; Thorne & Debener, ). This explanation might not seem parsimonious at first, but its plausibility is supported after recent findings of analogous asymmetries in synchrony processing across different modality orders in other electrophysiological studies (Kaganovich & Schumaker, ; Kösem et al., ).…”

Facilitation of Crossmodal Integration During Emotional Prediction in Methamphetamine Dependents

“…It has been suggested that the A-leading cues served as an alerting mechanism to boost visual processing immediately in crossmodal integration; in contrast, V-leading cues promoted the auditory system to make predictions about forthcoming auditory events (Thorne and Debener, 2014). Thus, our results suggested that auditory signs could activate the visual cortex faster to prepare for the incoming visual processing (Cecere et al, 2017) and improve the detection of facial emotions for MADs.…”

“…Under crossmodal integration, the anterior stimuli played the priming role (Stekelenburg and Vroomen, 2007), which prompted sensory systems to enter the readiness state and to speed up the response to later targets. Such prediction was likely related to the sensory modality of the cues (Cecere et al, 2017). Based on the prediction and promotion effect of the emotional cues during the crossmodal mode, Experiment 2 discussed whether the emotional recognition disorder of MADs could be improved through the designs of sequential presentation of crossmodal cues.…”

“…It is suggested that the prediction of fearful sound played an important role in the crossmodal of MADs (Wang et al, 2017). This asymmetry between the visual-and auditory-leading cues suggested that there were separate, selectively recruited networks (Cecere et al, 2017). It has been suggested that the A-leading cues served as an alerting mechanism to boost visual processing immediately in crossmodal integration; in contrast, V-leading cues promoted the auditory system to make predictions about forthcoming auditory events (Thorne and Debener, 2014).…”