Quantifying the contribution of vision to speech perception in noise

Abstract

The intelligibility of sentences presented in noise improves when the listener can view the talker's face. Our aims were to quantify this benefit, and to relate it to individual differences among subjects in lipreading ability and among sentences in lipreading difficulty. Auditory and audiovisual speech-reception thresholds (SRTs) were measured in 20 listeners with normal hearing. Sixty sentences, selected to range in the difficulty with which they could be lipread (with vision alone) from easy to hard, were presented for identification in white noise. Using the ascending method of limits, the SRT was defined as the lowest signal-to-noise ratio at which all three 'key words' in each sentence could be identified correctly. Measured as the difference in dB between auditory-alone and audiovisual SRTs, 'audiovisual benefit' averaged 11 dB, ranging from 6 to 15 dB among subjects, and from 3 to 22 dB among sentences. As predicted, audiovisual benefit is a measure of lipreading ability. It was highly correlated with visual-alone performance (n = 20, r = 0.86, P less than 0.01). Likewise, those sentences which were easiest to lipread gave a higher measure of benefit from vision in audiovisual conditions than did sentences that were hard to lipread (n = 60, r = 0.92, P less than 0.01). The results establish the basis of an efficient test of speech-reception disability in which measures are freed from the floor and ceiling effects encountered when percentage correct is used as the dependent variable.

When half a face is as good as a whole: Effects of simple substantial occlusion on visual and audiovisual speech perception

“…Pretesting had established that this signal-to-noise ratio produced approximately 70% correct responses to auditory stimuli presented without visual speech. This point on the performance scale was chosen to provide appropriate conditions for revealing effects of display type on the influences exerted by congruent and incongruent visual speech on auditory speech perception (see, e.g., Jordan & Sergeant, 1998;MacLeod & Summerfield, 1987, 1990Middleweerd & Plomp, 1987;Sumby & Pollack, 1954). All sound measurements were made with a decibelometer at the location of each participant.…”

“…The facial movements that accompany speech production (visual speech) are a powerful component of speech perception (e.g., Erber, 1969;Jordan, McCotter, & Thomas, 2000;MacLeod & Summerfield, 1987, 1990McGurk & MacDonald, 1976;Middleweerd & Plomp, 1987;Reisberg, McLean, & Goldfield, 1987;Rosenblum, Yakel, & Green, 2000;Sumby & Pollack, 1954;Thomas & Jordan, 2002). In particular, seeing the articulating face of a talker can improve auditory speech intelligibility substantially in quiet and noisy environments, and in the McGurk effect (after McGurk & MacDonald, 1976), can alter the perceived identity of speech sounds.…”

An analysis of British regional accent and contextual cue effects on speechreading performance

“…Finally, it should be noted that the speechreading performance scores obtained from our participants were rather low in comparison to most other studies using equivalent (BKB) stimulus materials. For instance, MacLeod & Summerfield (1987) found average performance ranged from around 20 to 50% of keywords, compared to the less than 10% obtained in the current study. This is a relatively common problem in tests of speechreading performance, with authors utilising various methods to improve performance; such as the inclusion of a low level of auditory speech (Landers & Davis, 2008).…”

“…'Mother baked a cake' and have been utilised in several studies of speechreading performance (e.g. MacLeod & Summerfield, 1987).…”

Using the visual component in automatic speech recognition

“…The benefit gained from the visual, facial cues has been quantitatively estimated to be equivalent to an increase of 8-10 dB in the signalto-noise ratio when speech sentences are presented in a noise background [9]. This observation suggests that, if the acoustic inputs to conventional speech recognition systems could be augmented by data about the visible speech gestures, an enhanced-performance, audio-visual recognition system should be possible.…”