Auditory-visual interactions

Abstract

Most displays or other devices designed to communicate information focus on a single sensory modality. This is perhaps a consequence of the fact that most research examining human perceptual capabilities has focused on a single sensory system at a time. However, most events in the natural environment generate physical information affecting multiple sensory modalities. This information is typically co-located in both space and time, and our perceptual systems have evolved to create a single coherent representation of our environment. Recent research has increasingly acknowledged the importance of these regularities, and the topic of multisensory integration became critical for understanding global awareness of the environment. This chapter presents an overview of the most recent research into the integration of auditory and visual information and presents considerations for the design of multisensory (i.e., auditory-visual) displays. The initial section of the chapter presents comparison of the basic features of the auditory and visual modalities and examines the effects of interacting visual and auditory stimuli. Included in this section is a discussion of the relative strengths and weaknesses of each modality and the situations in which one modality dominates. This section is followed by a discussion of perceptual effects when information from both modalities is complementary or conflicting and the role of multisensory attention on the perception of auditory-visual information. The final section of the chapter presents a discussion of the applications of auditory-visual integration principles to audiovisual display (e.g., helmet-mounted displays [HMDs]) and signal designs. Research on the interaction of a third modality, tactile or haptic displays, is just beginning and will not be discussed extensively here, although many of the same considerations can be expected to apply (see Chapter 18, Exploring the Tactile Modality for HMDs).

Auditory Spatial Perception: Auditory Localization

“…In general, if vision and hearing report conflicting information, vision almost always dominates hearing. This phenomenon has been termed the capture effect (e.g., Ghirardelli & Scharine, 2009). The most widely known form of the capture effect is the ventriloquism effect (VE) (Howard and Templeton, 1966) in which the listener perceives the ventriloquist's speech as coming from ventriloquist's dummy.…”

“…Many observations indicate that visual perception dominates auditory perception with respect to localization and that people have a tendency to trust their eyes more than their ears (Ghirardelli and Scharine, 2009;p. 605).…”