A Visual Speech Intelligibility Benefit Based on Speech Rhythm

Abstract

This study examined whether visual speech provides speech-rhythm information that perceivers can use in speech perception. This was tested by using speech that naturally varied in the familiarity of its rhythm. Thirty Australian English L1 listeners performed a speech perception in noise task with English sentences produced by three speakers: an English L1 speaker (familiar rhythm); an experienced English L2 speaker who had a weak foreign accent (familiar rhythm), and an inexperienced English L2 speaker who had a strong foreign accent (unfamiliar speech rhythm). The spoken sentences were presented in three conditions: Audio-Only (AO), Audio-Visual with mouth covered (AVm), and Audio-Visual (AV). Speech was best recognized in the AV condition regardless of the degree of foreign accent. However, speech recognition in AVm was better than AO for the speech with no foreign accent and with a weak accent, but not for the speech with a strong accent. A follow-up experiment was conducted that only used the speech with a strong foreign accent, under more audible conditions. The results also showed no difference between the AVm and AO conditions, indicating the null effect was not due to a floor effect. We propose that speech rhythm is conveyed by the motion of the jaw opening and closing, and perceivers use this information to better perceive speech in noise.

The Role of Talking Faces in Infant Language Learning: Mind the Gap between Screen-Based Settings and Real-Life Communicative Interactions

“…Given that auditory and visual cues provide overlapping information about the same speech event, they have often been named “redundant audiovisual (AV) speech cues” (e.g., [ 4 , 5 ]). Prior studies have shown that access to such AV redundant speech cues, compared to auditory-only situations, can facilitate lexical access and, more largely, speech comprehension, most notably when the acoustic signal becomes difficult to understand due to noise [ 6 , 7 , 8 , 9 , 10 , 11 ] or to an unfamiliar accent or language (e.g., [ 12 , 13 , 14 ]). In such occasions, adult listeners have been shown to increase their visual attention (hereafter attention) to the talker’s mouth in order to maximize the processing of AV speech cues and enhance their processing of speech; for instance, when background acoustic noise increases [ 15 , 16 ], when volume is low [ 17 ], when their language proficiency is low [ 18 , 19 , 20 ], or when they are performing particularly challenging speech-processing tasks (e.g., speech segmentation [ 21 ] or sentences comparison [ 18 ]).…”

Advances in Understanding the Phenomena and Processing in Audiovisual Speech Perception

“…Studies on the effects of different types of training and experience provide much-needed contributions to this knowledge. As far as long-term (lifetime) experience is concerned, Kawase, Davis and Kim [ 3 ] found that a familiar speech rhythm (when a speaker talks in their mother tongue) benefits AV speech perception in noise; meanwhile, an unfamiliar speech rhythm (when a speaker has a strong foreign accent) does not.…”