Voice pitch as an aid to lipreading

Abstract

The totally deafened adult, unable to make use of a hearing aid, has no alternative to lipreading for everyday communication. Lipreading, however, is no substitute for hearing speech. Many lipreaders have great difficulty in ideal conditions and even the best lipreaders find the task demanding and tiring. Prosthetic attempts to substitute for lost hearing have centred on three distinct types of intervention, visual, tactile, and electrocochlear. As none of these is likely to yield a good understanding of a speech independent of lipreading in the near future, we have attempted to isolate relatively simple patterns of stimulation that, although not intelligible in themselves, well aid lipreading. From this point of view, the fundamental frequency or 'pitch' of the voice is the most important pattern element because if provides both segmental and suprasegmental information and is practically invisible. It thus complements the visual information already available on the face. As we show here, with the voice pitch presented acoustically, normal listeners can lipread a speaker reading continuous text at up to two and a half times the rate possible on the basis of lipreading alone. The pitch signal by itself, of course, is completely unintelligible. Although our work is primarily concerned with methods of electrical stimulation of the cochlea, it has implications for other sensory substitution techniques, the design of special purpose hearing aids and current theories of speech perception.

Multimodal perceptual organization of speech: Evidence from tone analogs of spoken utterances

“…On the reports of prior research, we might have expected the analog of the first formant (Bernstein et al, 1992) or of the fundamental frequency of phonation (Rosen et al, 1981) to combine readily with the video samples in evoking an impression of the linguistic message. Why did the results of our test depart from the precedents?…”

The Role of Lip-reading and Cued Speech in the Processing of Phonological Information in French-educated Deaf Children

“…It has been shown repeatedly that nonspeech auditory signals combine with lip-read information to produce genuine phonetic percepts (e.g. Breeuwer & Plomp, 1986;Green & Miller, 1985;Rosen, Fourcin, & Moore, 1981).…”

Importance of Visual Cues in Hearing Restoration by Auditory Prosthesis

“…Such multi-modal processing results in fast and accurate recognition of the perceived signals. Speech perception effectively utilizes the visual information from the speaker's face not only in patients with hearing loss but also in healthy subjects; i.e., speech perception in degraded conditions such as background noise can be improved by visual information obtained from the speaker's face [12,15]. Therefore, visual cues (speaker's face) presented with auditory cues (speech sound) will be utilized to complement the auditory information in every situation.…”

“…10.1). The "lip-reading" phenomenon is well known in patients with degraded speech perception; i.e., reduced speech perception in the presence of poor auditory conditions, such as background noise and in patients with hearing loss, is improved by the combined presentation of visual speech [12,15]. If the degraded speech can be perceived as bimodal audiovisual stimuli, the visual information from the speaker's face can be effectively utilized to compensate for the inadequate auditory information [2,9,13].…”