Attentional resources contribute to the perceptual learning of talker idiosyncrasies in audiovisual speech

Abstract

To recognize audiovisual speech, listeners evaluate and combine information obtained from the auditory and visual modalities. Listeners also use information from one modality to adjust their phonetic categories to a talker's idiosyncrasy encountered in the other modality. In this study, we examined whether the outcome of this cross-modal recalibration relies on attentional resources. In a standard recalibration experiment in Experiment 1, participants heard an ambiguous sound, disambiguated by the accompanying visual speech as either /p/ or /t/. Participants' primary task was to attend to the audiovisual speech while either monitoring a tone sequence for a target tone or ignoring the tones. Listeners subsequently categorized the steps of an auditory /p/-/t/ continuum more often in line with their exposure. The aftereffect of phonetic recalibration was reduced, but not eliminated, by attentional load during exposure. In Experiment 2, participants saw an ambiguous visual speech gesture that was disambiguated auditorily as either /p/ or /t/. At test, listeners categorized the steps of a visual /p/-/t/ continuum more often in line with the prior exposure. Imposing load in the auditory modality during exposure did not reduce the aftereffect of this type of cross-modal phonetic recalibration. Together, these results suggest that auditory attentional resources are needed for the processing of auditory speech and/or for the shifting of auditory phonetic category boundaries. Listeners thus need to dedicate attentional resources in order to accommodate talker idiosyncrasies in audiovisual speech.

Do They Know It's Christmash? Lexical Knowledge Directly Impacts Speech Perception

“…Critically, this design required the use of a 4‐alternative forced choice (4AFC) task, wherein listeners were required to categorize both the final segment of the context item and the first segment of the target item. It is possible that this cognitively demanding 4AFC task obscured potential LCfC effects, especially since previous work suggests that some perceptual effects in speech processing may be attenuated when task demands are heightened; for example, increasing cognitive load can attenuate cross‐modal phonetic recalibration effects driven by visual (lipreading) information (Jesse & Kaplan, 2019), and shifting attentional resources away from the speech signal can extinguish the influence of lexical knowledge on phonetic retuning (Samuel, 2016). In addition to simple complexity/demand issues, note that McQueen and colleagues in various papers have argued that the Ganong effect results from postperceptual bias, while accepting that CfC is a perceptual‐level effect.…”