Dissociation of perception and action in audiovisual multisensory integration

Abstract

The “temporal rule” of multisensory integration proposes that unisensory stimuli, and the neuronal responses they evoke, must fall within a window of integration. Ecological validity demands that multisensory integration should occur only for physically simultaneous events (which may give rise to non-simultaneous neural activations), and spurious neural response simultaneities unrelated to environmental multisensory occurrences must somehow be rejected. Two experiments investigated the requirements of simultaneity for facilitative multisensory integration. Experiment 1 employed an RT/race model paradigm to measure audiovisual multisensory integration as a function of audiovisual stimulus onset asynchrony under fully dark adapted conditions for visual stimuli that were either rod- or cone-isolating. Auditory stimulus intensity was constant. Despite a 155 ms delay in mean RT to the scotopic versus photopic stimulus, facilitative audiovisual multisensory integration in both conditions occurred exclusively at an audiovisual stimulus onset asynchrony of 0 ms. Thus, facilitative multisensory integration demands both physical and physiological simultaneity. Experiment 2 investigated the accuracy of simultaneity and temporal order judgments under the same stimulus conditions. Judgments of audiovisual stimulus simultaneity or temporal order were significantly influenced by stimulus intensity indicating different simultaneity requirements for these tasks. We consider the possibility that there are mechanisms by which the nervous system may take account of variations in response latency arising from changes in stimulus intensity in order to selectively integrate only those physiological simultaneities that arise from physical simultaneities. We propose that separate subsystems for audiovisual multisensory integration exist which pertain to action and perception.

Audiovisual integration in depth: multisensory binding and gain as a function of distance

“…Overall, these results are reminiscent of findings by Leone and McCourt ( 2013 , 2015 ), who manipulated auditory and visual stimulus intensity, as well as stimulus timing, to measure the effects of physical and physiological simultaneity on multisensory gain. Using violations of the race model inequality as an indicator of multisensory gain, they found that despite significant differences in RT to unisensory stimuli that resulted from variations in stimulus intensity (i.e., more intense stimuli resulted in faster RTs), multisensory gain was restricted to a narrow range of SOAs within 50 ms of simultaneity (see also van der Stoep et al 2015b ).…”

“…Taken together, the results seemingly indicate that the SJ and MRT tasks likely index partially dependent multisensory processes that are somewhat dissociable and more readily indexed when considering relatively larger inter-subject variability (see Leone and McCourt 2015 ; Harrar et al 2016 , for a similar argument). In other words, while an opposing relationship between multisensory binding and gain as a function of distance is not necessarily evidenced in group-level measures, this relation becomes more apparent at an individual subject level (when controlling for intra-subject variability).…”

“…For those SOAs at which significant rMRE was observed, the cumulative distribution function (CDF) of RTs to A, V, and AV targets was calculated for each distance (near and far). As for the MRE analysis, the unisensory CDFs were first corrected for SOA (see Leone and McCourt 2015 ; Harrar et al 2016 ) and then summed to calculate the upper bound of statistical facilitation predicted by a race model (see Eq. 3 ; Colonius and Diederich 2017 ; Miller 1982 , 1986 , 2015 ; Raab 1962 ; Ulrich et al 2007 ).…”

Multisensory simultaneity judgment and proximity to the body

“…We suggest that in order to correctly bind information emanating from a common source, the neural processing scheme may allow for a greater temporal lead by the part of the sensory modality for which the native reference frame has been misaligned with respect to the body (and thus may require additional processing time). That is, we conjecture that the computation performed in order to make SJs may adapt as to correct for physiological lags due to reference frame transformation/reconciliation (see Leone & McCourt, 2013 , 2015 , for a similar argument). In the second experiment, we test this prediction by dissociating either the auditory or the visual reference frame with respect to the body-centered reference frame.…”

Crowdsourced Measurement of Reaction Times to Audiovisual Stimuli With Various Degrees of Asynchrony

“…This V shape is illustrated in Figure 1 , showing results from our literature survey on reaction times to audiovisual stimuli as a function of SOA. Only studies that used equivalent task conditions were included in this figure (for additional relevant research on SOA, see Harrar, Harris, & Spence, 2017 ; Leone & McCourt, 2015 ; Van der Stoep, Spence, Nijboer, & Van der Stigchel, 2015 ). Each subfigure shows mean reaction times as a function of SOA, where a negative SOA value means that the onset of the visual stimulus occurred after the onset of the auditory stimulus.…”