Fusion of Visual and Auditory Stimuli during Saccades: A Bayesian Explanation for Perisaccadic Distortions

Abstract

Brief stimuli presented near the onset of saccades are grossly mislocalized in space. In this study, we investigated whether the Bayesian hypothesis of optimal sensory fusion could account for the mislocalization. We required subjects to localize visual, auditory, and audiovisual stimuli at the time of saccades (compared with an earlier presented target). During fixation, vision dominates and spatially "captures" the auditory stimulus (the ventriloquist effect). But for perisaccadic presentations, auditory localization becomes more important, so the mislocalized visual stimulus is seen closer to its veridical position. The precision of the bimodal localization (as measured by localization thresholds or just-noticeable difference) was better than either the visual or acoustic stimulus presented in isolation. Both the perceived position of the bimodal stimuli and the improved precision were well predicted by assuming statistically optimal Bayesian-like combination of visual and auditory signals. Furthermore, the time course of localization was well predicted by the Bayesian approach. We present a detailed model that simulates the time-course data, assuming that perceived position is given by the sum of retinal position and a sluggish noisy eye-position signal, obtained by integrating optimally the output of two populations of neural activity: one centered at the current point of gaze, the other centered at the future point of gaze.

Abnormal Binaural Spectral Integration in Cochlear Implant Users

“…Studies of multi-input integration in other sensory modalities show that information is often averaged between multiple inputs. In the visual system, incongruent inputs to the two eyes are averaged such that information about the individual cues is lost in the integration process (Hillis et al 2002;Anstis and Roger 2012); similar effects are seen in auditory-visual integration for spatially incongruent auditory and visual stimuli (Binda et al 2007). Are similar averaging and loss of information effects also seen with binaural integration?…”

Reinforcing the Weight Attributed to Auditory Space across Saccades: Figure 1.

“…This model assumes that when a specific signal has high variance, the system gives it low reliability and, thus, decreases its weight at the time of creation of a unified percept (Ernst and Bülthoff, 2004). Authors report that sensory fusion is extremely well predicted by Bayesian inferences, supporting the notion that auditory and visual spatial cues are combined in a statistically optimal manner, where bimodal localization depends less on visual signals at the time of saccades than during fixation because of increased variance [Binda et al (2007) Another frequent situation inducing a decrease in visual saliency is when the same object appears in the peripheral visual field rather than in the fovea. Testing audiovisual integration in the central and the peripheral visual field would thus be interesting to observe whether the results obtained by Binda et al extend to all natural situations of visual impoverishment or are specific to precise situations, such as during saccades.…”

“…However, one may regret that, because of hardware constraints, the eccentricity of auditory stimuli could only be varied by steps of 6°, compared with the eccentricity of visual stimuli, which could be manipulated by steps of 1°. The use of more precise variations in the location of auditory sources in future studies would allow a more direct comparison between auditory and visual localization threshold, which in turn would certainly refine the conclusions of Binda et al (2007) regarding auditoryvisual interactions.…”

“…The authors found that, whereas the position of sounds had no influence on the localization of audiovisual targets during fixation, it significantly affected audiovisual spatial localization during saccades by reducing visual bias (Fig.1) [Binda et al (2007), their Fig. 2 (http://www.jneurosci.org/ cgi/content/full/27/32/8525/F2)].…”

“…In their recent article published in the Journal of Neuroscience, Binda et al (2007) explored this topic by cleverly using saccades, which occur more than three times per second in humans, as a natural cause of visual distortion. The authors took advantage of the fact that saccades have little influence on auditory space perception but distort visual space to explore how visual reliability affects auditory influence on audiovisual localization.…”