Importance of Visual Cues in Hearing Restoration by Auditory Prosthesis

Abstract

Auditory prostheses, such as cochlear implant and auditory brainstem implant, are used clinically to restore the hearing of patients with sensorineural hearing loss. These devices can considerably improve the auditory information conveyed to the auditory cortex, but proper rehabilitation process is usually necessary to restore auditory communication to an adequate level. Therefore, improvements in the auditory information provided by the prosthesis can be complemented by better rehabilitation process.Moreover, the complementary role of visual cues is also important. The "lipreading" phenomenon is well known in patients with degraded speech perception; i.e., reduced speech perception in the presence of poor auditory conditions, such as background noise and in patients with hearing loss, is improved by the combined presentation of visual speech. In addition to such conventional lip-reading, audiovisual speech has another beneficial role in the auditory rehabilitation process; i.e., the visual cue enhances the auditory adaptation process to the degraded speech sound.In the present paper, these two aspects of audio-visual speech in auditory rehabilitation are reviewed.

Mouth2Audio: intelligible audio synthesis from videos with distinctive vowel articulation

“…When we engage in face-to-face conversations, facial movements and corresponding voice are simultaneously used to perceive speech (Jongman et al, 2003;Kawase et al, 2015;Munhall et al, 2004).With multimedia (e.g., during video conferencing), we rely on visual facial cues when the audio is not transmitted well. Moreover, in noisy environments (e.g., cafeteria), seeing a talker's facial movements can particularly aid speech perception (Bernstein et al, 2004;Sumby & Pollack, 1954).…”