Audiovisual and lexical cues do not additively enhance perceptual adaptation

Abstract

When listeners experience difficulty in understanding a speaker, lexical and audiovisual (or lipreading) information can be a helpful source of guidance. These two types of information embedded in speech can also guide perceptual adjustment, also known as recalibration or perceptual retuning. With retuning or recalibration, listeners can use these contextual cues to temporarily or permanently reconfigure internal representations of phoneme categories to adjust to and understand novel interlocutors more easily. These two types of perceptual learning, previously investigated in large part separately, are highly similar in allowing listeners to use speech-external information to make phoneme boundary adjustments. This study explored whether the two sources may work in conjunction to induce adaptation, thus emulating real life, in which listeners are indeed likely to encounter both types of cue together. Listeners who received combined audiovisual and lexical cues showed perceptual learning effects similar to listeners who only received audiovisual cues, while listeners who received only lexical cues showed weaker effects compared with the two other groups. The combination of cues did not lead to additive retuning or recalibration effects, suggesting that lexical and audiovisual cues operate differently with regard to how listeners use them for reshaping perceptual categories. Reaction times did not significantly differ across the three conditions, so none of the forms of adjustment were either aided or hindered by processing time differences. Mechanisms underlying these forms of perceptual learning may diverge in numerous ways despite similarities in experimental applications.

Want to quickly adapt to distorted speech and become a better listener? Read lips, not text

“…Similar, but generally smaller assimilative aftereffects have been reported when instead of lipread speech, written text [ 17 ] or lexical information [ 16 , 18 – 20 ] is used as the teacher-signal that disambiguates the ambiguous phoneme. The reason why lipread speech is more potent than text or lexical information remains, for the time being, rather elusive.…”

Neural Correlates of Phonetic Adaptation as Induced by Lexical and Audiovisual Context

“…Listeners can draw on information other than the acoustic signal-lip movements or lexical/semantic knowledge-to adjust boundaries between speech sound categories so that they fit the speech input they are currently hearing, which enables them to adapt to pronunciations they have perhaps never heard. Behavioral evidence (Ullas et al, 2020b) suggests that, despite the apparent similarity, these two adaptation processes may have distinct triggers (coping with noise in the case of audiovisual recalibration and coping with talker novelty in the case of lexical retuning), although both types of adaptation often occur conjointly in real life. In this study, fMRI data were collected as participants underwent both forms of phoneme category adjustments, using lexical and audiovisual cues, respectively, in a counterbalanced, blocked design.…”

“…In audiovisual processing, the visual cues such as lip movements are available earlier to the listener ( Jesse & Massaro, 2010), and thus strong perceptual shifts can be observed after only a few exposure items, but these effects also diminish quickly ( Vroomen, van Linden, Keetels, de Gelder, & Bertelson, 2004), whereas lexical cues can lead to longer-lasting, more robust effects, but after long exposures toward one particular phoneme (Eisner & McQueen, 2006). When lexical and audiovisual effects are compared under the same exposure and testing conditions, with short exposures (i.e., eight biasing items) in alternation with short categorization tests on ambiguous items, both adaptation effects occur, with audiovisual cues generating larger perceptual shifts than lexical cues (Ullas, Formisano, Eisner, & Cutler, 2020a;van Linden & Vroomen, 2007); the behavioral effects are however not additive (Ullas, Formisano, Eisner, & Cutler, 2020b).…”

“…The ambiguous phoneme blends, to be perceived differently depending on the prior exposure block, can consist of either edited words or videos. The exposure time can also be matched; although lexical retuning studies typically use longer exposure phases to induce a bias, such retuning can take place in shorter time spans and can be observed in shorter test blocks, similar to the typical audiovisual exposure, as well (Ullas et al, 2020a(Ullas et al, , 2020bvan Linden & Vroomen, 2007).…”

The McGurk effect in the time of pandemic: Age-dependent adaptation to an environmental loss of visual speech cues

“…A question that remains is whether-even after access to all speech cues is regained-speech perception continues to be affected by the prior experience with altered availability of cues from the individual modalities. Aftereffects in multimodal speech perception are evidenced by studies on perceptual recalibration showing that (fully accessible) visual cues enable perceivers to disambiguate a speech sound intermediate between two categories and update the auditory content of those categories, which influences subsequent (auditory) speech-sound identification (Bertelson, Vroomen, & de Gelder, 2003;Ullas, Formisano, Eisner, & Cutler, 2020). Shifts in cue weighting brought about by a person's recent experience of persistently lower availability, and hence lower informativeness, of cues via a particular modality could also lead to perceptual adaptation-a decreased reliance on cues from that modality, even if they in fact are currently available in the speech signal.…”