Lip-Reading Aids Word Recognition Most in Moderate Noise: A Bayesian Explanation Using High-Dimensional Feature Space

Abstract

Watching a speaker's facial movements can dramatically enhance our ability to comprehend words, especially in noisy environments. From a general doctrine of combining information from different sensory modalities (the principle of inverse effectiveness), one would expect that the visual signals would be most effective at the highest levels of auditory noise. In contrast, we find, in accord with a recent paper, that visual information improves performance more at intermediate levels of auditory noise than at the highest levels, and we show that a novel visual stimulus containing only temporal information does the same. We present a Bayesian model of optimal cue integration that can explain these conflicts. In this model, words are regarded as points in a multidimensional space and word recognition is a probabilistic inference process. When the dimensionality of the feature space is low, the Bayesian model predicts inverse effectiveness; when the dimensionality is high, the enhancement is maximal at intermediate auditory noise levels. When the auditory and visual stimuli differ slightly in high noise, the model makes a counterintuitive prediction: as sound quality increases, the proportion of reported words corresponding to the visual stimulus should first increase and then decrease. We confirm this prediction in a behavioral experiment. We conclude that auditory-visual speech perception obeys the same notion of optimality previously observed only for simple multisensory stimuli.

Dissociable Effects of Aging and Mild Cognitive Impairment on Bottom-Up Audiovisual Integration

“…Adults with age-related deterioration in the auditory system will have familiarity with using other contextual cues (including lip-reading) to help disambiguate auditory speech information. Bayesian models of audiovisual speech integration suggest that the modality with less confusion should play a larger role for optimal integration [70,71], and behavioral studies confirm that information from the most reliable modality is given greater weight during perception of multisensory speech stimuli [72,73]. Supporting this view, the unisensory accuracy of both older groups was greater for identifying speech information in the visual modality than in the auditory modality.…”

Redundant spoken labels facilitate perception of multiple items

“…Although online effects of vision on audition are well known (Ma, Zhou, Ross, Foxe, & Parra, 2009;McGurk & MacDonald, 1976), less is known about the degree to which auditory information affects online visual processing and visual attention. There is now accumulating evidence that sounds affect visual perception (Sekuler, Sekuler, & Lau, 1997;Shams, Kamitani, & Shimojo, 2002), with modulations of early visual cortex by sounds detected in as little as 35-65 msec (Shams, Iwaki, Chawla, & Bhattacharya, 2005).…”

Multisensory Facilitation of Behavior in Monkeys: Effects of Stimulus Intensity

“…Interestingly, this profile was observed in 49% of the tested neurons. In the case of psychophysics, this principle is often applied to the observation of larger benefits of multisensory stimulation when the unisensory stimuli are themselves near threshold and/or noisy (Ma, Zhou, Ross, Foxe, & Parra, 2009;Bolognini, Leo, Passamonti, Stein, & Làdavas, 2007;Rach & Diederich, 2006;Diederich & Colonius, 2004;Grant & Seitz, 2000;Sumby & Pollack, 1954). This principle can also be applied when considering the perceptual benefits of multisensory interactions in the case of sensory deficits (Rouger et al, 2007;Laurienti et al, 2006;Hairston, Laurienti, Mishra, Burdette, & Wallace, 2003).…”