How actions shape perception: learning action-outcome relations and predicting sensory outcomes promote audio-visual temporal binding

Abstract

To maintain a temporally-unified representation of audio and visual features of objects in our environment, the brain recalibrates audio-visual simultaneity. This process allows adjustment for both differences in time of transmission and time for processing of audio and visual signals. In four experiments, we show that the cognitive processes for controlling instrumental actions also have strong influence on audio-visual recalibration. Participants learned that right and left hand button-presses each produced a specific audio-visual stimulus. Following one action the audio preceded the visual stimulus, while for the other action audio lagged vision. In a subsequent test phase, left and right button-press generated either the same audio-visual stimulus as learned initially, or the pair associated with the other action. We observed recalibration of simultaneity only for previously-learned audio-visual outcomes. Thus, learning an action-outcome relation promotes temporal grouping of the audio and visual events within the outcome pair, contributing to the creation of a temporally unified multisensory object. This suggests that learning action-outcome relations and the prediction of perceptual outcomes can provide an integrative temporal structure for our experiences of external events.

Perceiving your hand moving: BOLD suppression in sensory cortices and the role of the cerebellum in the detection of feedback delays

“…We did not find a behavioral advantage in bimodal conditions, which goes against previous findings on voluntary actions and multisensory processing (Arabzadeh et al, 2008;Desantis & Haggard, 2016b;Desantis, Mamassian, et al, 2014;van Kemenade et al, 2016). This could be explained by the fact that the auditory stimulus in the current study was not perceived as part of the movement because the visual feedback was naturalistic, whereas the auditory stimulus consisted of a beep, which was much more abstract.…”

“…Our actions shape the way we perceive the world around us (Blakemore, Wolpert, & Frith, 1998;Desantis & Haggard, 2016b;Haggard, Clark, & Kalogeras, 2002;Wolpert, Miall, & Kawato, 1998). For example, it has been well established that sensory stimuli associated with self-generated movements are experienced as less intense and produce less neural activity in corresponding sensory cortices than if they were externally delivered (Blakemore, Rees, & Frith, 1998;Roussel, Hughes, & Waszak, 2014;Shergill et al, 2013).…”

Irrelevant auditory and tactile signals, but not visual signals, interact with the target onset and modulate saccade latencies

“…In an earlier study, we found that the temporal integration of a beep and a flash can modulate the perceived appearance of the flash, so that it is perceived later or, to some extent, earlier in time, depending on whether it is preceded or followed by a beep [4]. Such asymmetrical modulation was also reported for multisensory integration after adaptation to an audiovisual temporal lag [35][36][37]. Here we found a similar asymmetrical modulations of saccade triggering, suggesting that the same multisensory mechanisms might be involved.…”

Sensorimotor temporal recalibration: the contribution of motor-sensory and inter-sensory components

“…This suggests that temporal recalibration is specific to the events that are causally-related. However, voluntary actions seem to provide us with an additional advantage in recalibrating temporal perception 15 . Actions can trigger an internal forward model which predicts the sensory consequences of the action based on the efference copy of the motor command 16–18 .…”

“…Such predictive processing may lead to stronger adaptation of motor-sensory pairings relative to purely sensory event pairings. Indeed, it has been consistently demonstrated that actions provide a temporal window into which binding of sensory events can be facilitated 12,15,19–21 .…”