The influence of static eye and head position on the ventriloquist effect

Abstract

Orienting responses to audiovisual events have shorter reaction times and better accuracy and precision when images and sounds in the environment are aligned in space and time. How the brain constructs an integrated audiovisual percept is a computational puzzle because the auditory and visual senses are represented in different reference frames: the retina encodes visual locations with respect to the eyes; whereas the sound localisation cues are referenced to the head. In the well-known ventriloquist effect, the auditory spatial percept of the ventriloquist's voice is attracted toward the synchronous visual image of the dummy, but does this visual bias on sound localisation operate in a common reference frame by correctly taking into account eye and head position? Here we studied this question by independently varying initial eye and head orientations, and the amount of audiovisual spatial mismatch. Human subjects pointed head and/or gaze to auditory targets in elevation, and were instructed to ignore co-occurring visual distracters. Results demonstrate that different initial head and eye orientations are accurately and appropriately incorporated into an audiovisual response. Effectively, sounds and images are perceptually fused according to their physical locations in space independent of an observer's point of view. Implications for neurophysiological findings and modelling efforts that aim to reconcile sensory and motor signals for goal-directed behaviour are discussed.

The principle of inverse effectiveness in audiovisual speech perception

“…First, in line with several laboratory studies of multisensory integration using simple sensory stimuli (e.g. white noise bursts and LED flashes) [16][17][18][19][20][21][22][23]25,26], a lower auditory SNR typically induced stronger multisensory enhancement. However, here we report that for the lowest SNRs (-21 dB) the enhancement saturated, or even slightly dropped (Fig.…”

“…speech-reading/lipreading) has a positive impact on speech perception, and audiovisual speech recognition in acoustic noise is substantially better than for auditory speech alone [4][5][6][7][8][9][10][11][12][13][14][15]. Audiovisual integration in general, has been the topic of a variety of behavioral and electrophysiological studies, involving rapid eyeorienting to simple peripheral stimuli [16,17], spatial and temporal discrimination of audiovisual objects [18][19][20], and the integrative responses of single neurons in cats and monkeys [21][22][23]. Three main principles have been shown to govern the mechanisms of multisensory integration: i. spatial alignment of the different sources, ii.…”

Predicting auditory space calibration from recent multisensory experience

“…Subjects had to avoid any head movements. Eye movements were not controlled for, since they have been shown to have no effect on the ventriloquism effect (Bertelson et al 2000 ; van Barneveld and van Wanrooij 2013 ). The time interval between the fixation light and stimulus onset varied randomly between 0.5 and 1 s.…”

Effect of extreme adaptive frequency compression in bimodal listeners on sound localization and speech perception

“…Head-movement signals were analyzed in MatLab (version 2014b) using custom-written software (Van Barneveld and Van Wanrooij, 2013). For each stimulus and participant, we determined the best-fit linear regression line between stimulus and response location, respectively α T and α R :…”