Want to quickly adapt to distorted speech and become a better listener? Read lips, not text

Abstract

When listening to distorted speech, does one become a better listener by looking at the face of the speaker or by reading subtitles that are presented along with the speech signal? We examined this question in two experiments in which we presented participants with spectrally distorted speech (4-channel noise-vocoded speech). During short training sessions, listeners received auditorily distorted words or pseudowords that were partially disambiguated by concurrently presented lipread information or text. After each training session, listeners were tested with new degraded auditory words. Learning effects (based on proportions of correctly identified words) were stronger if listeners had trained with words rather than with pseudowords (a lexical boost), and adding lipread information during training was more effective than adding text (a lipread boost). Moreover, the advantage of lipread speech over text training was also found when participants were tested more than a month later. The current results thus suggest that lipread speech may have surprisingly long-lasting effects on adaptation to distorted speech.

Perceptual Adaptation to Noise-Vocoded Speech by Lip-Read Information: No Difference between Dyslexic and Typical Readers

“…In this study with typical readers, it was found that accuracy on audiovisual training items was ∼65%, whereas participants whoinstead of dynamic lip-read information -received a static image of the speaker during training only identified ∼21% of the words correctly. Moreover, accuracy on auditory-only test items (presented without any visual information) increased with ∼20% when comparing the first with the last test block for participants in the audiovisual lip-read training condition, which starkly contrasted the ∼3% increase in performance for participants who had received a static image of the speaker's face during the audiovisual training blocks (Pourhashemi et al, 2022). In the current study, we only compared, for dyslexic and typical readers, the extent to which lip-read speech boosts spoken-word recognition (i.e., a lip-read advantage of audiovisual training blocks over auditory-only test blocks), and the extent to which listeners adapt to noise-vocoded speech over time (i.e., an increase in performance in subsequent training and test blocks), so without a condition with a static face.…”

“…The stimulus set contained 120 audiovisual recordings of mono-and bisyllablic words (see also Pourhashemi et al, 2022, where the full list of items is provided). A male native Dutch speaker (one of the authors: MB) pronounced the words while being recorded by a Nikon D7200 camera and an external microphone attached to the camera (Røde VideoMicro).…”

“…This disambiguating context can generate an error signal that, in turn, drives changes in the acoustic-phonetic mapping (for a review, see Guediche et al, 2014). For example, adaptation is enhanced when distorted speech is (1) accompanied by a written version of the speech signal (Francis et al, 2007;Loebach et al, 2010;Schwab et al, 1985), (2) preceded by a clear undistorted acoustic version of the speech input (Davis et al, 2005;Hervais-Adelman et al, 2008), or (3) combined with lip-read speech (Pourhashemi et al, 2022). In all these cases, listeners rely on the disambiguating context to more readily adapt to the acoustic distortions and, as a result, thus become better listeners.…”

“…To examine this question, we used the stimuli and design of Pourhashemi et al (2022) where multiple audio-only test blocks were interspersed with audiovisual training blocks. Each block contained 15 unique auditory distorted words that were not previously encountered in the task.…”

Repeatedly experiencing the McGurk effect induces long-lasting changes in auditory speech perception

“…Our results clarify that FIR, if induced appropriately, can persist over timescales even longer than those described for LGP. A recent comparison between the benefits of visual speech and written text in understanding degraded auditory speech showed that visual speech provided a greater benefit than written speech, and this benefit persisted for more than a month, supporting the idea of a longlasting influence of visual speech on auditory speech perception 44 .…”