Bayesian Calibration of Simultaneity in Audiovisual Temporal Order Judgments

Abstract

After repeated exposures to two successive audiovisual stimuli presented in one frequent order, participants eventually perceive a pair separated by some lag time in the same order as occurring simultaneously (lag adaptation). In contrast, we previously found that perceptual changes occurred in the opposite direction in response to tactile stimuli, conforming to Bayesian integration theory (Bayesian calibration). We further showed, in theory, that the effect of Bayesian calibration cannot be observed when the lag adaptation was fully operational. This led to the hypothesis that Bayesian calibration affects judgments regarding the order of audiovisual stimuli, but that this effect is concealed behind the lag adaptation mechanism. In the present study, we showed that lag adaptation is pitch-insensitive using two sounds at 1046 and 1480 Hz. This enabled us to cancel lag adaptation by associating one pitch with sound-first stimuli and the other with light-first stimuli. When we presented each type of stimulus (high- or low-tone) in a different block, the point of simultaneity shifted to “sound-first” for the pitch associated with sound-first stimuli, and to “light-first” for the pitch associated with light-first stimuli. These results are consistent with lag adaptation. In contrast, when we delivered each type of stimulus in a randomized order, the point of simultaneity shifted to “light-first” for the pitch associated with sound-first stimuli, and to “sound-first” for the pitch associated with light-first stimuli. The results clearly show that Bayesian calibration is pitch-specific and is at work behind pitch-insensitive lag adaptation during temporal order judgment of audiovisual stimuli.

How actions shape perception: learning action-outcome relations and predicting sensory outcomes promote audio-visual temporal binding

“…This suggests that recalibration of audio-visual simultaneity is conditional on executing the action that participants had previously learned would produce the audio-visual pair, and was not dependent on the audio-visual pair itself. This corroborates recent studies showing that the identity of the audio-visual pair does not drive temporal recalibration827 (but see ref. 25).…”

Encoding of event timing in the phase of neural oscillations

“…Nevertheless, the latter finding remains consistent with other lag-adaptation reports (Miyazaki et al, 2006;Yamamoto et al, 2012) and major differences in experimental design including the choice of task and the absence of readapting trials in the TOJ assessment task could account for these differences (Cai et al, 2012;Yamamoto et al, 2012). Specifically, remote and recent stimulation histories are known to bias in opposite ways the perception of incoming stimuli (Chopin and Mamassian, 2012) and consistent with this, PSS shifts are distinctly influenced by the presence or the absence of re-adapting trials (Cai et al, 2012).…”

“…Shifts in perceived AV simultaneity following lag-adaptation (Fujisaki et al, 2004;Miyazaki et al, 2006;Vroomen et al, 2004;Yamamoto et al, 2012) have been hypothesized to originate from mechanisms capable of adjusting the neural processing time across sensory modalities (Fujisaki et al, 2004;Moutoussis and Zeki, 1997;Stone et al, 2001;Sugita and Suzuki, 2003;Zeki and Bartels, 1998). In support of this hypothesis, our study reveals that such mechanisms may be implemented as phase shifts of neural oscillations: contrasting the sensory responses before and after AV lag-adaptation provided no evidence for a latency code hypothesis and instead revealed significant phase shifts of the entrained 1 Hz neural oscillations.…”

Intersensory binding across space and time: A tutorial review

“…Miyazaki, Yamamoto, Uchida and Kitazawa (2006) hypothesized that Bayesian calibration is at work during judgments regarding audiovisual temporal order but that the effect is concealed behind lag adaptation mechanism (Miyazaki et al, 2006). By canceling lag adaptation by using two pitches of sounds, they successfully uncovered "Bayesian calibration" that was working behind lag adaptation (Yamamoto, Miyazaki, Iwano & Kitazawa, 2012). In a recent study, Aihara (2009, 2011) proposed a unifying Bayesian model to account for temporal ventriloquism aftereffects.…”