High visual resolution matters in audiovisual speech perception, but only for some

Abstract

The basis for individual differences in the degree to which visual speech input enhances comprehension of acoustically degraded speech is largely unknown. Previous research indicates that fine facial detail is not critical for visual enhancement when auditory information is available; however, these studies did not examine individual differences in ability to make use of fine facial detail in relation to audiovisual speech perception ability. Here, we compare participants based on their ability to benefit from visual speech information in the presence of an auditory signal degraded with noise, modulating the resolution of the visual signal through lowpass spatial frequency filtering and monitoring gaze behavior. Participants who benefited most from the addition of visual information (high visual gain) were more adversely affected by the removal of high spatial frequency information, compared to participants with low visual gain, for materials with both poor and rich contextual cues (i.e., words and sentences, respectively). Differences as a function of gaze behavior between participants with the highest and lowest visual gains were observed only for words, with participants with the highest visual gain fixating longer on the mouth region. Our results indicate that the individual variance in audiovisual speech in noise performance can be accounted for, in part, by better use of fine facial detail information extracted from the visual signal and increased fixation on mouth regions for short stimuli. Thus, for some, audiovisual speech perception may suffer when the visual input (in addition to the auditory signal) is less than perfect.

Audio-visual integration in noise: Influence of auditory and visual stimulus degradation on eye movements and perception of the McGurk effect

“…When monologues were presented in high levels of background noise including music and multilingual talkers, participants looked at the eyes approximately half of the time (Vatikiotis-Bateson, Eigsti, Yano & Munhall, 1998 ). It could be argued that this is due to the nature and length of the stimuli (45 s) as participants may be looking for social/emotional cues whilst listening to the narrative (Alsius, Wayne, Paré & Munhall, 2016 ). Another study found that participants focused more on the nose and mouth when sentences were presented in noise (multi-talker babble), again suggesting that the area directly surrounding the mouth is important (Buchan, Paré & Munhall, 2008 ).…”

Visual perception of vowels from static and dynamic cues

“…As noted by a reviewer this may have provided salient features and this may have influenced the perception. For example, previous studies have shown that viewers increase their gaze at the mouth region when images have increased resolution (e.g., Alsius et al, 2016), however, this strategy does not necessarily lead to improved performance. Although the participants in this study did not report that the dots inhibited or facilitated their decision, it cannot be precluded that their performance would have been more or less successful without the dots.…”

Looking Behavior and Audiovisual Speech Understanding in Children With Normal Hearing and Children With Mild Bilateral or Unilateral Hearing Loss

“…Visual information also could have been processed through peripheral channels, which are very important for audiovisual speech perception. For example, perceptual studies using spatially filtered audiovisual speech stimuli have shown that lower spatial frequencies are the main contributors to audiovisual speech perception (Munhall, Kroos et al 2004), although there are individual differences in sensitivity to high spatial frequency information in speech (Alsius, et al 2016). Furthermore, studies of audiovisual speech integration have shown that the McGurk effect (McGurk & MacDonald 1976) persists even when the visual stimulus is up to 40° in the periphery (Paré, et al 2003).…”