Auditory-Visual Interactions Subserving Goal-Directed Saccades in a Complex Scene

Abstract

This study addresses the integration of auditory and visual stimuli subserving the generation of saccades in a complex scene. Previous studies have shown that saccadic reaction times (SRTs) to combined auditory-visual stimuli are reduced when compared with SRTs to either stimulus alone. However, these results have been typically obtained with high-intensity stimuli distributed over a limited number of positions in the horizontal plane. It is less clear how auditory-visual interactions influence saccades under more complex but arguably more natural conditions, when low-intensity stimuli are embedded in complex backgrounds and distributed throughout two-dimensional (2-D) space. To study this problem, human subjects made saccades to visual-only (V-saccades), auditory-only (A-saccades), or spatially coincident auditory-visual (AV-saccades) targets. In each trial, the low-intensity target was embedded within a complex auditory-visual background, and subjects were allowed over 3 s to search for and foveate the target at 1 of 24 possible locations within the 2-D oculomotor range. We varied systematically the onset times of the targets and the intensity of the auditory target relative to background [i.e., the signal-to-noise (S/N) ratio] to examine their effects on both SRT and saccadic accuracy. Subjects were often able to localize the target within one or two saccades, but in about 15% of the trials they generated scanning patterns that consisted of many saccades. The present study reports only the SRT and accuracy of the first saccade in each trial. In all subjects, A-saccades had shorter SRTs than V-saccades, but were more inaccurate than V-saccades when generated to auditory targets presented at low S/N ratios. AV-saccades were at least as accurate as V-saccades but were generated at SRTs typical of A-saccades. The properties of AV-saccades depended systematically on both stimulus timing and S/N ratio of the auditory target. Compared with unimodal A- and V-saccades, the improvements in SRT and accuracy of AV-saccades were greatest when the visual target was synchronous with or leading the auditory target, and when the S/N ratio of the auditory target was lowest. Further, the improvements in saccade accuracy were greater in elevation than in azimuth. A control experiment demonstrated that a portion of the improvements in SRT could be attributable to a warning-cue mechanism, but that the improvements in saccade accuracy depended on the spatial register of the stimuli. These results agree well with earlier electrophysiological results obtained from the midbrain superior colliculus (SC) of anesthetized preparations, and we argue that they demonstrate multisensory integration of auditory and visual signals in a complex, quasi-natural environment. A conceptual model incorporating the SC is presented to explain the observed data.

Catching audiovisual mice: Predicting the arrival time of auditory-visual motion signals

“…In the latter case, the bimodal performance is limited by the visual system, and the additional auditory input does not substantially increase the performance. The results are consistent with those reported by Corneil et al (2002), using a very different experimental paradigm and task. Corneil et al measured saccadic eye movements in response to auditory, visual, and bimodal stimuli; they found that the accuracy of the bimodally driven saccades is determined largely by the accuracy of the visually initiated saccades and not by the auditory ones.…”

“…On the other hand, if the object is hidden for a short period (Condition 2) and the task therefore requires the extrapolation of motion speed over a temporal and spatial period, the facilitation due to both sensory inputs is almost absent, and the bimodal performance is limited by the visual performance. rooij, Munoz, & Van Opstal, 2002) corroborated previous behavioral and electrophysiological findings (Stein & Meredith, 1993) by demonstrating that the improvement in SRTs is optimal when the auditory and the visual stimuli are synchronized. EEG studies, in combination with behavioral RT measurements (Molholm et al, 2002), have confirmed these behavioral findings by demonstrating shorter bimodal RTs than unimodal ones; this facilitation can be accounted for by neural summation, rather than by an independent decision model (race model).…”

Intersensory binding across space and time: A tutorial review

“…When audiovisual stimuli are spatially aligned within~20°of vertical separation and observers have to orient to an auditory or a visual target, both latency and accuracy improve, relative to the unisensory and misaligned conditions. This thus indicates a rule of "best-of-both-worlds" (Corneil, Van Wanrooij, Munoz & Van Opstal, 2002), in which observers benefit from the spatial accuracy provided by the visual component, and a shorter latency onset that is triggered by the auditory component. Prior expectations about the audiovisual spatial alignment also matter: That is, participants are faster on aligned trials when 100 % of the trials are aligned, rather than only 10 % of the trials (Van Wanrooij, Bremen & Van Opstal, 2010), thus suggesting that audiovisual binding may have a dynamic component that depends on the evidence for stimulus congruency as acquired from prior experience.…”

Bimodal and trimodal multisensory enhancement: Effects of stimulus onset and intensity on reaction time

“…This may be one reason why it has not shown up in most behavioral studies with humans. Corneil, Van Wanrooij, Munoz, and Van Opstal (2002) suggested some further reasons. First, in many studies, the stimuli may not have been close enough to threshold.…”