Cross-modal correspondence between auditory pitch and visual elevation modulates audiovisual temporal recalibration

Abstract

Cross-modal correspondences refer to associations between feature dimensions of stimuli across sensory modalities. Research has indicated that correspondence between audiovisual stimuli influences whether these stimuli are integrated or segregated. On the other hand, the audiovisual integration process plastically changes to compensate for continuously observed spatiotemporal conflicts between sensory modalities. If and how cross-modal correspondence modulates the “recalibration” of integration is unclear. We investigated whether cross-modal correspondence between auditory pitch and visual elevation affected audiovisual temporal recalibration. Participants judged the simultaneity of a pair of audiovisual stimuli after an adaptation phase in which alternating auditory and visual stimuli equally spaced in time were presented. In the adaptation phase, auditory pitch and visual elevation were manipulated to fix the order within each pairing of audiovisual stimuli congruent with pitch-elevation correspondence (visual leading or auditory leading). We found a shift in the point of subjective simultaneity (PSS) between congruent audiovisual stimuli as a function of the adaptation conditions (Experiment 1, 2), but this shift in the PSS was not observed within incongruent pairs (Experiment 2). These results indicate that asynchronies between audiovisual signals congruent with cross-modal correspondence are selectively recalibrated.

Malleability and fluidity of time perception

“…Gamma-aminobutyric acid (GABA) levels in the human visual cortex measured using magnetic resonance spectroscopy appear to correlate with perceived durations of visual intervals, suggesting that the GABAergic system contributes to individual differences in time perception 29 . However, time perception studies in this Collection have found that learning of temporal interval discrimination transfers between auditory and visual modalities 30 and that cross-modal correspondence between auditory pitch and visual elevation selectively affects temporal recalibration 31 .…”

Investigating Crossmodal Correspondences Between Vibrotactile Stimuli and Colors

“…Over recent years scholars were able to identify and verify existing crossmodal correspondences, with a special focus put on the investigation of the correspondence between auditory and visual stimuli. Thus, the crossmodal correspondence between sound pitch (audition) and object shape (vision) exemplified by the "Kiki/Bouba" effect (Ramachandran et al, 2003;Ćwiek et al, 2021), sound pitch (audition) and object elevation in space (vision) (Uno et al, 2022), as well as between both pitch and loudness (audition) and lightness and brightness (vision) (Marks, 1987) were proven to exist. This research builds upon existing studies exploring the correlation between vibrotactile stimuli and color choice.…”

The effect of semantic content on the perception of audiovisual movieclips

“…The findings suggest that in all conditions, relative timing of auditory and visual stimuli was not a critical factor for enhanced visual sensitivity in detecting actions, but semantic congruency was (Thomas & Shiffrar, 2013). Recently, Uno & Yokosawa (2022) took into consideration the spatiotemporal congruency effects between modalities, and examined their potential relation with temporal recalibration of audiovisual stimuli during perceptual integration processing. For this purpose, they conducted simultaneity judgment tasks consisting of audiovisual pairs (audio pitch either high or low, and visual cycle either presented above fixation point or below) that were synchronous or asynchronous.…”