Audiovisual spoken word training can promote or impede auditory-only perceptual learning: prelingually deafened adults with late-acquired cochlear implants versus normal hearing adults

Abstract

Training with audiovisual (AV) speech has been shown to promote auditory perceptual learning of vocoded acoustic speech by adults with normal hearing. In Experiment 1, we investigated whether AV speech promotes auditory-only (AO) perceptual learning in prelingually deafened adults with late-acquired cochlear implants. Participants were assigned to learn associations between spoken disyllabic C(=consonant)V(=vowel)CVC non-sense words and non-sense pictures (fribbles), under AV and then AO (AV-AO; or counter-balanced AO then AV, AO-AV, during Periods 1 then 2) training conditions. After training on each list of paired-associates (PA), testing was carried out AO. Across all training, AO PA test scores improved (7.2 percentage points) as did identification of consonants in new untrained CVCVC stimuli (3.5 percentage points). However, there was evidence that AV training impeded immediate AO perceptual learning: During Period-1, training scores across AV and AO conditions were not different, but AO test scores were dramatically lower in the AV-trained participants. During Period-2 AO training, the AV-AO participants obtained significantly higher AO test scores, demonstrating their ability to learn the auditory speech. Across both orders of training, whenever training was AV, AO test scores were significantly lower than training scores. Experiment 2 repeated the procedures with vocoded speech and 43 normal-hearing adults. Following AV training, their AO test scores were as high as or higher than following AO training. Also, their CVCVC identification scores patterned differently than those of the cochlear implant users. In Experiment 1, initial consonants were most accurate, and in Experiment 2, medial consonants were most accurate. We suggest that our results are consistent with a multisensory reverse hierarchy theory, which predicts that, whenever possible, perceivers carry out perceptual tasks immediately based on the experience and biases they bring to the task. We point out that while AV training could be an impediment to immediate unisensory perceptual learning in cochlear implant patients, it was also associated with higher scores during training.

Multisensory training can promote or impede visual perceptual learning of speech stimuli: visual-tactile vs. visual-auditory training

“…This principle was also demonstrated in Bernstein et al (2014, Exp. 1), in which prelingually deaf adults whose higher rank system for speech perception is vision received paired-associates training with the goal of improving unisensory auditory perception.…”

Auditory Perception and Production of Speech Feature Contrasts by Pediatric Implant Users

“…This further suggests, however, that the addition of vision plays a role in the auditory learning of some distinctions because the children tended to be better able to perceive more visible distinctions through AO with greater accuracy than those that are less visible. The cross-modal value of audition and vision in auditory learning is consistent with the findings of Bernstein et al (2014). Eisenberg et al (2003) have further suggested that vision may provide sensory assistance as the auditory cortex continues to adapt to auditory input.…”

Sound localization in a changing world

“…These findings suggest that it might be beneficial to combine visual and auditory cues in the rehabilitation of the hearing impaired. However, audiovisual training appears to impede auditory speech learning in pre-lingually deaf individuals who receive cochlear implants later in life, presumably because of their greater dependence on visual speech perception than normalhearing subjects [87]. Nevertheless, a different result has been obtained in adult ferrets fitted with bilateral cochlear implants following deafening in infancy [14].…”