Bayesian integration of visual and auditory signals for spatial localization

Abstract

Human observers localize events in the world by using sensory signals from multiple modalities. We evaluated two theories of spatial localization that predict how visual and auditory information are weighted when these signals specify different locations in space. According to one theory (visual capture), the signal that is typically most reliable dominates in a winner-take-all competition, whereas the other theory (maximum-likelihood estimation) proposes that perceptual judgments are based on a weighted average of the sensory signals in proportion to each signal's relative reliability. Our results indicate that both theories are partially correct, in that relative signal reliability significantly altered judgments of spatial location, but these judgments were also characterized by an overall bias to rely on visual over auditory information. These results have important implications for the development of cue integration and for neural plasticity in the adult brain that enables humans to optimally integrate multimodal information.

Visual determinants of a cross-modal illusion

“…However, when the quality of the auditory information was degraded, vision did have a significant influence. Similar effects have been reported by Battaglia, Jacobs, and Aslin (2003) as well as by Alais and Burr (2004). Therefore, there is reason to believe that the data presented by Schutz and Lipscomb (2007) might be explained by the theory of optimal integration.…”

Causality and cross-modal integration.

“…However, when the quality of the auditory information was degraded, vision did have a significant influence. Similar effects have been reported by Battaglia, Jacobs, and Aslin (2003) as well as Alais and Burr (2004). Although optimal integration applies to a wide variety of sensory conflicts (Ernst & Banks, 2002), here we focus our review primarily on the relevant auditory-visual examples.…”

Generative Probabilistic Modeling: Understanding Causal Sensorimotor Integration

“…Psychophysics experiments (e.g., Alais & Burr, 2004;Battaglia, Jacobs, & Aslin, 2003) typically test multisensory perception for optimality by matching to the ideal-observer performance in two ways. First, the optimal estimate of the true height isŷ h,v = μ y|h,v , so the variance of the human's responsesŷ h,v to multisensory stimuli should match the variance of the optimal response σ 2 y|h,v (Eq.…”

“…Therefore, experimentally manipulating the variances σ 2 h , σ 2 v of the individual modalities should produce the appropriate changes in the human perceptual responseŷ h,v . These quantities can be determined directly in direct-estimation experiments (e.g., Wallace et al, 2004) or indirectly via fitting a psychometric function in two-alternative forced-choice experiments (e.g., Alais & Burr, 2004;Battaglia et al, 2003).…”