The use of visible speech cues for improving auditory detection of spoken sentences

Abstract

Classic accounts of the benefits of speechreading to speech recognition treat auditory and visual channels as independent sources of information that are integrated fairly early in the speech perception process. The primary question addressed in this study was whether visible movements of the speech articulators could be used to improve the detection of speech in noise, thus demonstrating an influence of speechreading on the ability to detect, rather than recognize, speech. In the first experiment, ten normal-hearing subjects detected the presence of three known spoken sentences in noise under three conditions: auditory-only (A), auditory plus speechreading with a visually matched sentence (AV(M)) and auditory plus speechreading with a visually unmatched sentence (AV(UM). When the speechread sentence matched the target sentence, average detection thresholds improved by about 1.6 dB relative to the auditory condition. However, the amount of threshold reduction varied significantly for the three target sentences (from 0.8 to 2.2 dB). There was no difference in detection thresholds between the AV(UM) condition and the A condition. In a second experiment, the effects of visually matched orthographic stimuli on detection thresholds was examined for the same three target sentences in six subjects who participated in the earlier experiment. When the orthographic stimuli were presented just prior to each trial, average detection thresholds improved by about 0.5 dB relative to the A condition. However, unlike the AV(M) condition, the detection improvement due to orthography was not dependent on the target sentence. Analyses of correlations between area of mouth opening and acoustic envelopes derived from selected spectral regions of each sentence (corresponding to the wide-band speech, and first, second, and third formant regions) suggested that AV(M) threshold reduction may be determined by the degree of auditory-visual temporal coherence, especially between the area of lip opening and the envelope derived from mid- to high-frequency acoustic energy. Taken together, the data (for these sentences at least) suggest that visual cues derived from the dynamic movements of the fact during speech production interact with time-aligned auditory cues to enhance sensitivity in auditory detection. The amount of visual influence depends in part on the degree of correlation between acoustic envelopes and visible movement of the articulators.

How Visual Cues for when to Listen Aid Selective Auditory Attention

“…Consistent with this explanation, the visual cue provided, if anything, a greater benefit for spatially separated sources when presented binaurally, which had the lowest threshold overall, than when presented diotically. The magnitude of the benefit we observed, which was on the order of a few decibels, is in the range found by other studies examining the benefits of visual cues for the detection and identification of speech (e.g., Sumby and Pollack 1954;Grant and Seitz 2000;Helfer and Freyman 2005).…”

“…However, temporal uncertainty seems to have a larger effect on performance when target and masker are more similar, such as when a masker is tonal rather than noise (Bonino and Leibold 2008). Moreover, both speech detection (e.g., Grant and Seitz 2000;Bernstein et al 2004;Tye-Murray et al 2011) and speech intelligibility (e.g., Sumby and Pollack 1954;Helfer and Freyman 2005) are enhanced when a visual stimulus depicting appropriate lip movements accompanies speech masked by noise. Bernstein et al (2004) noted that simpler visual cues, such as the presence of an abstract shape whose size follows the broadband envelope of the target speech or even a static shape whose appearance coincides with the target presentation, can also improve speech-in-noise detection thresholds.…”

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

“…Moreover, the auditory modality will eventually dominate speech perception when visual information is distorted [10,24]. Conversely, noise and auditory compression artifacts can mask essential spectral frequencies and lead to distracting audio effects [23]; in speech perception, reduced audio quality typically leads to an increased dependency on the visual modality [13]. Clearly, severe quality drops will make information from either modality ambiguous; however, within the boundaries of what is intelligible, temporal integration may not be equally affected.…”

Multisensory anatomical pathways

“…Stein and Meredith, 1993). Such multisensory processing can affect a range of different behavioral parameters, such as reaction times (Welch and Warren, 1986;Raab, 1962), stimulus detection rate (Grant and Seitz, 2000), accuracy of stimulus identification (Giard and Peronnet, 1999) as well as learning effects on stimulus processing Lehmann and Murray, 2005). While for example the decrease in reaction times under multisensory conditions have been largely reported for human subjects in auditory-visual recognition tasks, no behavioral data to our knowledge are available in monkeys performing similar protocols.…”