The impact of automatic exaggeration of the visual articulatory features of a talker on the intelligibility of spectrally distorted speech

Abstract

Visual speech information plays a key role in supporting speech perception, especially when acoustic features are distorted or inaccessible. Recent research suggests that for spectrally distorted speech, the use of visual speech in auditory training improves not only subjects' audiovisual speech recognition, but also their subsequent auditory-only speech recognition. Visual speech cues, however, can be affected by a number of facial visual signals that vary across talkers, such as lip emphasis and speaking style. In a previous study, we enhanced the visual speech videos used in perception training by automatically tracking and colouring a talker's lips. This improved the subjects' audiovisual and subsequent auditory speech recognition compared with those who were trained via unmodified videos or audio-only methods. In this paper, we report on two issues related to automatic exaggeration of the movement of the lips/ mouth area. First, we investigate subjects' ability to adapt to the conflict between the articulation energy in the visual signals and the vocal effort in the acoustic signals (since the acoustic signals remained unexaggerated). Second, we have examined whether or not this visual exaggeration can improve the subjects' performance of auditory and audiovisual speech recognition when used in perception training. To test this concept, we used spectrally distorted speech to train groups of listeners using four different training regimes: (1) audio only, (2) audiovisual, (3) audiovisual visually exaggerated, and (4) audiovisual visually exaggerated and lip-coloured. We used spectrally distorted speech (cochlear-implant-simulated speech) because the longer-term aim of our work is to employ these concepts in a training system for cochlear-implant (CI) users. The results suggest that after exposure to visually exaggerated speech, listeners had the ability to adapt alongside the conflicting audiovisual signals. In addition, subjects trained with enhanced visual cues (regimes 3 and 4) achieved better audiovisual recognition for a number of phoneme classes than those who were trained with unmodified visual speech (regime 2). There was no evidence of an improvement in the subsequent audio-only listening skills, however. The subjects' adaptation to the conflicting audiovisual signals may have slowed down auditory perceptual learning, and impeded the ability of the visual speech to improve the training gains. 1 Highlights. 1. An automatic method that exaggerates the visual articulatory features of talkers in audiovisual speech is presented. 2. An audiovisual conflict after-effect was found, but it becomes negligible after exposing subjects to exaggerated speech. 3. Exaggeration of the visual speech improves the audiovisual recognition of a number of phoneme classes. 4. Audiovisual recalibration to visually exaggerated speech may have impeded learning when used in the audiovisual training.

Audio-visual speech comprehension in noise with real and virtual speakers

“…Rosenblum, Johnson and Saldaña (1996) found that the speech recognition improved during the second half of the experiment for all audiovisual conditions, including video showing the full face as well as point light representations of the face while no such improvement was observed for the audio-only group. Adaptation effects are possibly more pronounced in the context of more unfamiliar audiovisual stimuli as suggested by a study by Alghamdi, Maddock, Barker, and Brown (2017) in which strong adaptation effects were observed for stimuli with exaggerated lip movements over repeated (training) sessions in a sentence recognition task. Furthermore, perceptual adaptation can even influence performance of subsequent similar but not identical perceptual tasks, like the 'McGurk aftereffect' as demonstrated by Bertelson, Vroomen, and De Gelder (2003).…”

“…Is there an adaptation effect involved in exploiting visual speech cues from a virtual speaker resulting in improved speech comprehension in background multitalker babble noise? If adaptation effects observed in studies of speech recognition (Alghamdi et al, 2017;Rosenblum et al, 1996) can be generalized to speech comprehension, we predict that there will be an increasing positive effect on speech comprehension (in background multitalker babble noise) by repeated exposure to the virtual speaker.…”

“…It could, however, provide stronger or more salient visual cues while filtering out potentially distracting natural occurring visual noise in the animation process. Previous studies have found some support for the potential to emphasize and exaggerate certain visual features to improve or investigate audiovisual speech processing (Alghamdi et al, 2017) and within the production of animated film, the strategy of emphasizing basic emotional expressions and key gestures by exaggeration is manifested as one of the 12 basic principles of animation (Lasseter, 1987;Thomas and Johnston, 1995). Using a virtual speaker like the one in the current study also has the potential advantageas a research toolto be able to modulate specific movement or features without necessarily drawing explicit attention to them (as might be a risk when presenting stylized abstractions of facial movement or edited video recordings).…”

PTeacher: a Computer-Aided Personalized Pronunciation Training System with Exaggerated Audio-Visual Corrective Feedback

“…Exaggerations in the form-focused instruction [85] have been verified to be beneficial for inexperienced L2 learners. Alghamdi et al [1] investigated that exaggeration of the visual speech improved the audio-visual recognition of many phoneme classes. Exaggeration methods were used in CAPT systems to assist L2 learners in perceiving stress patterns.…”

“…In offline English classes, exaggerated feedback has been a effective feedback strategy for teachers to rectify the pronunciation of learners [62,73]. Alghamdi, et al [1] have proven that visually exaggerated speech is capable of promoting the perceptual ability of learners. However, this direction has rarely been explored in the field of human-computer interactions.…”