Rapid recalibration to audiovisual asynchrony follows the physical—not the perceived—temporal order

Abstract

In natural scenes, audiovisual events deriving from the same source are synchronized at their origin. However, from the perspective of the observer, there are likely to be significant multisensory delays due to physical and neural latencies. Fortunately, our brain appears to compensate for the resulting latency differences by rapidly adapting to asynchronous audiovisual events by shifting the point of subjective synchrony (PSS) in the direction of the leading modality of the most recent event. Here we examined whether it is the perceived modality order of this prior lag or its physical order that determines the direction of the subsequent rapid recalibration. On each experimental trial, a brief tone pip and flash were presented across a range of stimulus onset asynchronies (SOAs). The participants' task alternated over trials: On adaptor trials, audition either led or lagged vision with fixed SOAs, and participants judged the order of the audiovisual event; on test trials, the SOA as well as the modality order varied randomly, and participants judged whether or not the event was synchronized. For test trials, we showed that the PSS shifted in the direction of the physical rather than the perceived (reported) modality order of the preceding adaptor trial. These results suggest that rapid temporal recalibration is determined by the physical timing of the preceding events, not by one's prior perceptual decisions.

Sensory- and memory-related drivers for altered ventriloquism effects and aftereffects in older adults

“…Serial dependencies in perceptual decisions can be driven by multiple factors, such as the previous stimuli (Bruns & Röder, 2015;Kayser & Kayser, 2018;Wozny & Shams, 2011b), the previous responses (Fritsche et al, 2017;Kiyonaga et al, 2017;Talluri et al, 2018), or a meta-cognitive assessment of previous behaviour (Benwell et al, 2019). Yet, previous work in young participants considered the ventriloquism aftereffect generally as a sensory-driven phenomenon Van der Burg et al, 2018). Confirming this view, we found that the sensory contribution to the aftereffect prevailed in the YA.…”

The neurophysiological basis of short- and long-term ventriloquism aftereffects

“…Indeed, the two prominent events during the AV trial are the exposure to a discrepant audiovisual stimulus and the corresponding response. Contrasting these two in their predictive power of the subsequent bias, we previously reported in favor of a sensory-rather than motorrelated origin of the ventriloquism aftereffect Park et al 2020;Van der Burg et al 2018). In the present dataset we found that the previous motor response contributes significantly, in addition to the multisensory discrepancy.…”

“…Previous work has suggested that adaptive recalibration, in particular on a trial-by-trial level, may not only be related to the previous sensory input but also to variables related to the reported experience, or previous motor response (Park et al 2020;Van der Burg et al 2018).…”

Shared neural underpinnings of multisensory integration and trial-by-trial perceptual recalibration

“…the previous multisensory discrepancy: Δ n-1 = V n-1 – A n-1 ) or the participant’s response in that trial (R n-1 ). Formal model comparison revealed that the model R n ~ 1 + A n + Δ n-1 provided a better account of the data than a response-based model R n ~ 1 + A n + R n-1 (relative BIC: 0, 393; BIC weights: 1, 0), supporting the notion that recalibration is linked more to the physical stimuli than the participants response (21).…”

“…However, we interact with our environment using sequences of actions dealing with different stimuli, and thus systematic sensory discrepancies as required for long-term effects are possibly seldom encountered. Hence, while the behavioral patterns of multisensory trial-by-trial recalibration are frequently studied (6,7,9,20,21) it remains unclear when and where during sensory processing their neural underpinnings emerge.…”