The context of experienced sensory discrepancies shapes multisensory integration and recalibration differently

Abstract

Whether two sensory cues interact during perceptual judgments depends on their immediate properties, but as suggested by Bayesian models, also on the observer's a priori belief that these originate from a common source. While in many experiments this a priori belief is considered fixed, in real life it must adapt to the momentary context or environment. To understand the adaptive nature of human multisensory perception we investigated the context-sensitivity of spatial judgements in a ventriloquism paradigm. We exposed observers to audio-visual stimuli whose discrepancy either varied over a wider (±46°) or a narrower range (±26°) and hypothesized that exposure to a wider range of discrepancies would facilitate multisensory binding by increasing participants a priori belief about a common source for a given discrepancy. Our data support this hypothesis by revealing an enhanced integration (ventriloquism) bias in the wider context, which was echoed in Bayesian causal inference models fit to participants' data, which assigned a stronger a priori integration tendency during the wider context. Interestingly, the immediate ventriloquism aftereffect, a multisensory response bias obtained following a multisensory test trial, was not affected by the contextual manipulation, although participants' confidence in their spatial judgments differed between contexts for both integration and recalibration trials. These results highlight the context-sensitivity of multisensory binding and suggest that the immediate ventriloquism aftereffect is not a purely sensory-level consequence of the multisensory integration process.

Different time scales of common-cause evidence shape multisensory integration, recalibration and motor adaptation

“…Despite the hand and computer cursor moving in different spatial planes, both signals are perceptually combined and the presentation of visually rotated feedback engages the processes of multisensory integration (Ladwig et al, 2013; Rand and Heuer, 2013; Debats et al, 2017b; Kirsch and Kunde, 2019, 2022), multisensory recalibration(Rand and Heuer, 2019b), and motor adaptation (Krakauer et al, 2000; Bock et al, 2003; Debats et al, 2023). In fact, the properties of multisensory integration and recalibration seen in this visuo‐motor paradigm are analogous to those observed in purely perceptual paradigms such as the audio‐visual ventriloquism effect and aftereffect (Chen and Vroomen, 2013; Bruns, 2019; Park and Kayser, 2022). The likely reason for the perceptual combination of visual and proprioceptive signals in the visuo‐motor paradigm is that the spatially discrepant felt movements of the hand and seen movements of the cursor are temporally correlated in the same way as felt and seen movements of our hand.…”

“…Multisensory integration is known to be sensitive to the overall experimental context in which it is probed, and the bias observed in particular trials can depend on the nature of preceding trials in an experiment (Gau and Noppeney, 2016; Rand and Heuer, 2018; Rohe et al, 2019). For example, with a larger range of intermodal discrepancies we observed weaker integration in the visuo‐motor task (Debats et al, 2023), but stronger integration in an audio‐visual ventriloquism paradigm (Park and Kayser, 2022). In line with such a context dependency of integration we have previously observed that prolonged exposure to constant discrepancies between hand movements and visual feedback reduces the integration bias (Rand and Heuer, 2020).…”

“…The feedback marker had the format of a cloud of 100 grey dots (1 mm diameter each) the spatial coordinates of which had a bivariate normal distribution with standard deviations of 2 mm for both axes (truncated at three standard deviations so that the total diameter was about 12 mm). Visual feedback in the format of a cloud of small dots is only rarely used in visuo‐motor paradigms (Körding and Wolpert, 2004; Debats et al, 2021, 2023) but frequently in audio‐visual paradigms (Park et al, 2011; Rohe and Noppeney, 2015a; Beierholm et al, 2020; Park and Kayser, 2021, 2022) because it offers straightforward means to vary the reliability of the visual position information.…”

Cumulative multisensory discrepancies shape the ventriloquism aftereffect but not the ventriloquism bias

“…Most datasets analysed here have been published previously, though there are three exceptions (see Table 1 for details). Overall the data are from 11 experiments, all of which were designed to probe the ventriloquism effect and the immediate ventriloquism aftereffect following an established design (Wozny and Shams, 2011) that we have used in a number of previous studies (Park and Kayser, 2019, 2020, 2021; Park et al, 2021; Park and Kayser, 2022). Participants were adult volunteers who reported no history of neurological diseases, normal vision and hearing.…”

“…We analysed data from 10 experiments, eight of which have been published previously (see Table 1 for details). All experiments followed the same overall design template to probe the ventriloquism effect and the immediate aftereffect following established work (Wozny and Shams, 2011b; Park and Kayser, 2019, 2020, 2021; Park et al, 2021; Park and Kayser, 2022). Participants were adult volunteers who reported no history of neurological diseases, and normal vision and hearing.…”