Unifying multisensory signals across time and space

Abstract

The brain integrates information from multiple sensory modalities and, through this process, generates a coherent and apparently seamless percept of the external world. Although multisensory integration typically binds information that is derived from the same event, when multisensory cues are somewhat discordant they can result in illusory percepts such as the "ventriloquism effect." These biases in stimulus localization are generally accompanied by the perceptual unification of the two stimuli. In the current study, we sought to further elucidate the relationship between localization biases, perceptual unification and measures of a participant's uncertainty in target localization (i.e., variability). Participants performed an auditory localization task in which they were also asked to report on whether they perceived the auditory and visual stimuli to be perceptually unified. The auditory and visual stimuli were delivered at a variety of spatial (0 degrees, 5 degrees, 10 degrees, 15 degrees ) and temporal (200, 500, 800 ms) disparities. Localization bias and reports of perceptual unity occurred even with substantial spatial (i.e., 15 degrees ) and temporal (i.e., 800 ms) disparities. Trial-by-trial comparison of these measures revealed a striking correlation: regardless of their disparity, whenever the auditory and visual stimuli were perceived as unified, they were localized at or very near the light. In contrast, when the stimuli were perceived as not unified, auditory localization was often biased away from the visual stimulus. Furthermore, localization variability was significantly less when the stimuli were perceived as unified. Intriguingly, on non-unity trials such variability increased with decreasing disparity. Together, these results suggest strong and potentially mechanistic links between the multiple facets of multisensory integration that contribute to our perceptual Gestalt.

Generative Probabilistic Modeling: Understanding Causal Sensorimotor Integration

“…The second key aspect of this task which must be included in any full generative model of this problem is that oddity can be entailed in the probe stimulus not only by its combined difference from the standard, but by discrepancy within the probe stimulus. In this case (similar to other recent multisensory perception experiments with variable causal structure : Hairston et al, 2003;Shams et al, 2000Shams et al, , 2005Wallace et al, 2004), the variable structure can effectively "give away" the probe. We introduced the approach needed to solve this type of problem in multisensory perception as structure inference (Hospedales & Vijayakumar, 2008).…”

“…That is, it was not obvious which of multiple possible sources caused the observations (Hairston et al, 2003;Shams, Kamitani, & Shimojo, 2000;Shams, Ma, & Beierholm, 2005;Wallace et al, 2004), or which of multiple possible world models was true (Knill, 2007). In these cases, the standard MLI linear-cue-combination approach fails to explain human performance.…”

“…Therefore, experimentally manipulating the variances σ 2 h , σ 2 v of the individual modalities should produce the appropriate changes in the human perceptual responseŷ h,v . These quantities can be determined directly in direct-estimation experiments (e.g., Wallace et al, 2004) or indirectly via fitting a psychometric function in two-alternative forced-choice experiments (e.g., Alais & Burr, 2004;Battaglia et al, 2003).…”

“…We introduced the approach needed to solve this type of problem in multisensory perception as structure inference (Hospedales & Vijayakumar, 2008). Körding, Beierholm et al (2007) carried out a detailed analysis of the experiments of Hairston et al (2003) and Wallace et al (2004) and showed how the structure inference approach was necessary to explain the results, but they termed the procedure causal inference (see Chapter 2).…”

Perception of visual-tactile colocation in the first year of life.

“…Research with human adults and animals has shown that when (and when not) stimuli to different sense modalities originate from a common location in external space, this has important implications for neural processing and behaviour (e.g., Meredith & Stein, 1996;Stein & Stanford, 2008;Spence & Driver, 2004;Wallace, Roberson, Hairston, Stein, Vaughan, & Schirillo, 2004). Adults perceive and make use of spatial commonalities across the senses in a seemingly effortless manner.…”