Speech identification in noise: Contribution of temporal, spectral, and visual speech cues

Abstract

This study investigated the degree to which two types of reduced auditory signals (cochlear implant simulations) and visual speech cues combined for speech identification. The auditory speech stimuli were filtered to have only amplitude envelope cues or both amplitude envelope and spectral cues and were presented with/without visual speech. In Experiment 1, IEEE sentences were presented in quiet and noise. For in-quiet presentation, speech identification was enhanced by the addition of both spectral and visual speech cues. Due to a ceiling effect, the degree to which these effects combined could not be determined. In noise, these facilitation effects were more marked and were additive. Experiment 2 examined consonant and vowel identification in the context of CVC or VCV syllables presented in noise. For consonants, both spectral and visual speech cues facilitated identification and these effects were additive. For vowels, the effect of combined cues was underadditive, with the effect of spectral cues reduced when presented with visual speech cues. Analysis indicated that without visual speech, spectral cues facilitated the transmission of place information and vowel height, whereas with visual speech, they facilitated lip rounding, with little impact on the transmission of place information.

Hearing Speech in Noise: Seeing a Loud Talker is Better

“…An explanation for the finding that PC1 was the motion component that showed a significant correlation with the percent correct in noise AV scores is that the PC1 movements provided information about the auditory speech signal that could be used to parse this signal from the competing background noise (see Davis and Kim 2004;Kim et al 2009). To determine whether this was the case, a correlation analysis was conducted to examine the degree to which PC1 and the wideband intensity speech envelope (see figure 5) was correlated for in quiet and in noise productions.…”

Perceptual Doping: An Audiovisual Facilitation Effect on Auditory Speech Processing, From Phonetic Feature Extraction to Sentence Identification in Noise

“…The perceptual doping effect in the AV1–A2 modality order may have been so strong that it greatly helped the listeners to decode the temporal cues necessary to discriminate vowel duration in the vowel duration discrimination task and to extract phonological cues for vowel identification in the gated vowel identification task, subsequently boosting the participants’ performance on these tasks in the A modality. The addition of V cues might have a stronger effect for consonants than vowels in terms of their AV identification (Kim et al 2009; Moradi et al 2017a), which could explain why the abovementioned effect for vowels was not observed in the gated consonant task. For instance, Moradi et al (2017a) reported that the effect of adding V cues on AV identification and cognitive demand reduction was more evident for consonants than for vowels (i.e., more V saliency for the AV identification of consonants than vowels).…”

Speech comprehension aided by multiple modalities: Behavioural and neural interactions

“…Everyday speech comprehension is multi-faceted: in face-to-face conversation, the listener receives information from the voice and face of a talker, the accompanying gestures of the hands and body and the overall semantic context of the discussion, which can all be used to aid comprehension of the spoken message. Behaviourally, auditory speech comprehension is enhanced by simultaneous presentation of a face or face-like visual cues (Sumby and Pollack, 1954; Grant & Seitz, 2000b; Girin et al, 2001; Kim & Davis, 2004; Bernstein et al, 2004; Schwartz et al, 2004; Helfer & Freyman, 2005; Ross et al, 2007; Thomas & Pilling, 2007; Bishop & Miller, 2009; Kim et al, 2009; Ma et al, 2009; Hazan et al, 2010). Higher-order linguistic information can also benefit intelligibility: words presented in a sentence providing a rich semantic context are more intelligible than words in isolation or in an abstract sentence, particularly when auditory clarity is compromised (Miller & Isard, 1963; Kalikow et al, 1977; Pichora-Fuller et al, 1995; Dubno et al, 2000; Grant & Seitz, 2000a; Stickney & Assmann, 2001, Obleser et al, 2007).…”