Auditory speech detection in noise enhanced by lipreading

Bernstein, Lynne E.; Auer, Edward T.; Takayanagi, Sumiko

doi:10.1016/j.specom.2004.10.011

Cited by 171 publications

(153 citation statements)

References 25 publications

Supporting

Mentioning

142

Contrasting

Order By: Relevance

“…Is this just because we are disposed to perceive a simple correspondence between the diameter of a probable sound source and its amplitude? This cannot be a critical feature, as the identification of speech in noise is not helped by the perception of an annulus whose diameter is controlled by the amplitude of the acoustic signal (also see Bernstein et al 2004b;Ghazanfar et al 2005). This (among other demonstrations) suggests that, for the most efficient speech-reading, mouth opening and closing and the tongue position should be clearly visible.…”

Section: The Visible Speech Stream: Varieties Of Informationmentioning

confidence: 99%

The processing of audio-visual speech: empirical and neural bases

Campbell

2007

Phil. Trans. R. Soc. B

244

213

View full text Add to dashboard Cite

In this selective review, I outline a number of ways in which seeing the talker affects auditory perception of speech, including, but not confined to, the McGurk effect. To date, studies suggest that all linguistic levels are susceptible to visual influence, and that two main modes of processing can be described: a complementary mode, whereby vision provides information more efficiently than hearing for some under-specified parts of the speech stream, and a correlated mode, whereby vision partially duplicates information about dynamic articulatory patterning.Cortical correlates of seen speech suggest that at the neurological as well as the perceptual level, auditory processing of speech is affected by vision, so that 'auditory speech regions' are activated by seen speech. The processing of natural speech, whether it is heard, seen or heard and seen, activates the perisylvian language regions (leftOright). It is highly probable that activation occurs in a specific order. First, superior temporal, then inferior parietal and finally inferior frontal regions (leftOright) are activated. There is some differentiation of the visual input stream to the core perisylvian language system, suggesting that complementary seen speech information makes special use of the visual ventral processing stream, while for correlated visual speech, the dorsal processing stream, which is sensitive to visual movement, may be relatively more involved.

show abstract

Section: The Visible Speech Stream: Varieties Of Informationmentioning

confidence: 99%

The processing of audio-visual speech: empirical and neural bases

Campbell

2007

Phil. Trans. R. Soc. B

244

213

View full text Add to dashboard Cite

show abstract

“…Moreover, both speech detection (e.g., Grant and Seitz 2000;Bernstein et al 2004;Tye-Murray et al 2011) and speech intelligibility (e.g., Sumby and Pollack 1954;Helfer and Freyman 2005) are enhanced when a visual stimulus depicting appropriate lip movements accompanies speech masked by noise. Bernstein et al (2004) noted that simpler visual cues, such as the presence of an abstract shape whose size follows the broadband envelope of the target speech or even a static shape whose appearance coincides with the target presentation, can also improve speech-in-noise detection thresholds. These results suggest that visual cues can provide cross-modal enhancement of auditory perception in some energetic masking conditions (Stein and Meredith 1993).…”

Section: Introductionmentioning

confidence: 99%

How Visual Cues for when to Listen Aid Selective Auditory Attention

Varghese

Ozmeral

Best

et al. 2012

JARO

View full text Add to dashboard Cite

Visual cues are known to aid auditory processing when they provide direct information about signal content, as in lip reading. However, some studies hint that visual cues also aid auditory perception by guiding attention to the target in a mixture of similar sounds. The current study directly tests this idea for complex, nonspeech auditory signals, using a visual cue providing only timing information about the target. Listeners were asked to identify a target zebra finch bird song played at a random time within a longer, competing masker. Two different maskers were used: noise and a chorus of competing bird songs. On half of all trials, a visual cue indicated the timing of the target within the masker. For the noise masker, the visual cue did not affect performance when target and masker were from the same location, but improved performance when target and masker were in different locations. In contrast, for the chorus masker, visual cues improved performance only when target and masker were perceived as coming from the same direction. These results suggest that simple visual cues for when to listen improve target identification by enhancing sounds near the threshold of audibility when the target is energetically masked and by enhancing segregation when it is difficult to direct selective attention to the target. Visual cues help little when target and masker already differ in attributes that enable listeners to engage selective auditory attention effectively, including differences in spectrotemporal structure and in perceived location.

show abstract

“…Although we might not be aware of the relevance of the visual cues for normal speech recognition, the influence of vision becomes convincingly apparent when the auditory information is embedded in noise. In degraded auditory conditions, the visuoauditory presentation leads to higher performance of recognition, when compared with the auditory alone stimulation (8,9), in a mechanism that mimics an improvement in the acoustic signal-to-noise ratio (SNR) (10).…”

mentioning

confidence: 99%

Evidence that cochlear-implanted deaf patients are better multisensory integrators

Rouger

Lagleyre

Fraysse

et al. 2007

Proc. Natl. Acad. Sci. U.S.A.

252

276

View full text Add to dashboard Cite

The cochlear implant (CI) is a neuroprosthesis that allows profoundly deaf patients to recover speech intelligibility. This recovery goes through long-term adaptative processes to build coherent percepts from the coarse information delivered by the implant. Here we analyzed the longitudinal postimplantation evolution of word recognition in a large sample of CI users in unisensory (visual or auditory) and bisensory (visuoauditory) conditions. We found that, despite considerable recovery of auditory performance during the first year postimplantation, CI patients maintain a much higher level of word recognition in speechreading conditions compared with normally hearing subjects, even several years after implantation. Consequently, we show that CI users present higher visuoauditory performance when compared with normally hearing subjects with similar auditory stimuli. This better performance is not only due to greater speechreading performance, but, most importantly, also due to a greater capacity to integrate visual input with the distorted speech signal. Our results suggest that these behavioral changes in CI users might be mediated by a reorganization of the cortical network involved in speech recognition that favors a more specific involvement of visual areas. Furthermore, they provide crucial indications to guide the rehabilitation of CI patients by using visually oriented therapeutic strategies.cochlear implant ͉ deafness ͉ multisensory integration ͉ speech comprehension D espite the apparent division between sensory modalities from the receptors to high cortical levels, we can simultaneously integrate visual and auditory signals resulting in qualitative percepts distinct from those derived from a single unisensory stimulus (1, 2). Furthermore, in cases of precise temporal or spatial congruency between the bisensory stimuli, multisensory integration is expressed at the behavioral level by perceptual improvements by reducing ambiguity (3, 4) and at the neuronal level by enhancing neuronal activity (5). Multisensory integration is also essential for speech recognition, which is based on the simultaneous integration of visual information derived from lip movements and auditory cues produced by the talker (6). The McGurk effect, in which a mismatch between the visual and auditory speech signals is artificially introduced, reveals that the visual information derived from lip movements can strongly influence our auditory perception (7). Although we might not be aware of the relevance of the visual cues for normal speech recognition, the influence of vision becomes convincingly apparent when the auditory information is embedded in noise. In degraded auditory conditions, the visuoauditory presentation leads to higher performance of recognition, when compared with the auditory alone stimulation (8, 9), in a mechanism that mimics an improvement in the acoustic signal-to-noise ratio (SNR) (10).In normally hearing (NH) subjects, although speechreading performance is very low, the association during development between the...

show abstract

Auditory speech detection in noise enhanced by lipreading

Cited by 171 publications

References 25 publications

The processing of audio-visual speech: empirical and neural bases

The processing of audio-visual speech: empirical and neural bases

How Visual Cues for when to Listen Aid Selective Auditory Attention

Evidence that cochlear-implanted deaf patients are better multisensory integrators

Contact Info

Product

Resources

About