Visual speech alters the discrimination and identification of non-intact auditory speech in children with hearing loss

Abstract

Objectives Understanding spoken language is an audiovisual event that depends critically on the ability to discriminate and identify phonemes yet we have little evidence about the role of early auditory experience and visual speech on the development of these fundamental perceptual skills. Objectives of this research were to determine 1) how visual speech influences phoneme discrimination and identification; 2) whether visual speech influences these two processes in a like manner, such that discrimination predicts identification; and 3) how the degree of hearing loss affects this relationship. Such evidence is crucial for developing effective intervention strategies to mitigate the effects of hearing loss on language development. Methods Participants were 58 children with early-onset sensorineural hearing loss (CHL, 53% girls, M = 9;4 yrs) and 58 children with normal hearing (CNH, 53% girls, M = 9;4 yrs). Test items were consonant-vowel (CV) syllables and nonwords with intact visual speech coupled to non-intact auditory speech (excised onsets) as, for example, an intact consonant/rhyme in the visual track (Baa or Baz) coupled to non-intact onset/rhyme in the auditory track (/–B/aa or /–B/az). The items started with an easy-to-speechread /B/ or difficult-to-speechread /G/ onset and were presented in the auditory (static face) vs. audiovisual (dynamic face) modes. We assessed discrimination for intact vs. non-intact different pairs (e.g., Baa:/–B/aa). We predicted that visual speech would cause the non-intact onset to be perceived as intact and would therefore generate more same—as opposed to different—responses in the audiovisual than auditory mode. We assessed identification by repetition of nonwords with non-intact onsets (e.g., /–B/az). We predicted that visual speech would cause the non-intact onset to be perceived as intact and would therefore generate more Baz—as opposed to az— responses in the audiovisual than auditory mode. Results Performance in the audiovisual mode showed more same responses for the intact vs. non-intact different pairs (e.g., Baa:/–B/aa) and more intact onset responses for nonword repetition (Baz for/–B/az). Thus visual speech altered both discrimination and identification in the CHL—to a large extent for the /B/ onsets but only minimally for the /G/ onsets. The CHL identified the stimuli similarly to the CNH but did not discriminate the stimuli similarly. A bias-free measure of the children’s discrimination skills (i.e., d’ analysis) revealed that the CHL had greater difficulty discriminating intact from non-intact speech in both modes. As the degree of HL worsened, the ability to discriminate the intact vs. non-intact onsets in the auditory mode worsened. Discrimination ability in CHL significantly predicted their identification of the onsets—even after variation due to the other variables was controlled. Conclusions These results clearly established that visual speech can fill in non-intact auditory speech, and this effect, in turn, made the non-intact onsets more difficult t...

Performance of Children With Hearing Loss on an Audiovisual Version of a Nonword Repetition Task

“…First, such a task offers ecological validity. In typical, everyday interactions, children often have access to visual information and cues along with the speech signal, and this access improves speech perception in children with hearing loss (Jerger, Damian, McAlpine, & Abdi, 2017) and CNH (Lalonde & Holt, 2016). Thus, the ability to process spoken language in the auditory-visual modality may be a better predictor of overall language learning potential.…”

Effects of Exogenous Auditory Attention on Temporal and Spectral Resolution

“…The sensitivity index d ′ was computed using MATLAB software for each subject and each condition (valid, invalid, and uncued) with the formula d ′ = z (hit rate) − z (false alarm) ( Macmillan and Creelman, 1991 ), which indicates the z-transformed probabilities of hit and false alarm rates. Based on the previous studies which applied SDT, the loglinear correction was used to adjust the hit rates and false alarm rates, whereby 0.5 was added to the number of hits and the number of false alarms, and 1 was added to both the number of signal trials and the number of noise trials ( Hautus, 1995 ; Stanislaw and Todorov, 1999 ; Lopez-Poveda et al, 2010 ; Lago et al, 2015 ; Jerger et al, 2017 ). This approach was used to take the non-finite values into account, that is, the extreme values of hit rates and false alarm rates.…”

Developmental Shifts in Detection and Attention for Auditory, Visual, and Audiovisual Speech

“…Testing was carried out within a double-walled soundtreated booth. The data of this study were gathered in one session of a multiple-day experimental protocol (e.g., Jerger et al, 2014;Jerger, Damian, Parra, & Abdi, 2017;Jerger, Damian, Tye-Murray, & Abdi, 2016. The presentation order of the facial conditions was counterbalanced across participants in each age group.…”