Interleaved lexical and audiovisual information can retune phoneme boundaries

Abstract

To adapt to situations in which speech perception is difficult, listeners can adjust boundaries between phoneme categories using perceptual learning. Such adjustments can draw on lexical information in surrounding speech, or on visual cues via speechreading. In the present study, listeners proved they were able to flexibly adjust the boundary between two plosive/stop consonants, /p/-/t/, using both lexical and speech-reading information and given the same experimental design for both cue types. Videos of a speaker pronouncing pseudo-words and audio recordings of Dutch words were presented in alternating blocks of either stimulus type. Listeners were able to switch between cues to adjust phoneme boundaries, and resulting effects were comparable to results from listeners receiving only a single source of information. Overall, audiovisual cues (i.e., the videos) produced the stronger effects, commensurate with their applicability for adapting to noisy environments. Lexical cues were able to induce effects with fewer exposure stimuli and a changing phoneme bias, in a design unlike most prior studies of lexical retuning. While lexical retuning effects were relatively weaker compared to audiovisual recalibration, this discrepancy could reflect how lexical retuning may be more suitable for adapting to speakers than to environments. Nonetheless, the presence of the lexical retuning effects suggests that it may be invoked at a faster rate than previously seen. In general, this technique has further illuminated the robustness of adaptability in speech perception, and offers the potential to enable further comparisons across differing forms of perceptual learning.

Neural Correlates of Phonetic Adaptation as Induced by Lexical and Audiovisual Context

“…The perceptual boundary between two phonemes, /p/ and /t/, was systematically shifted, using lexical and audiovisual cues, toward either /p/ or /t/. Note that the behavioral results had shown that this procedure resulted in significant effects in both conditions and toward both phonemes, although audiovisual recalibration effects were larger than lexical retuning, in line with previous findings as well (Ullas et al, 2020a;van Linden & Vroomen, 2007). The analysis of concurrent fMRI measurements showed similarities between audiovisual and lexical exposure blocks, particularly in the temporal cortex across bilateral HG, STG/STS, and PT as well as left IPL and right insula.…”

“…The phoneme bias of the exposure block alternated every two blocks. Although this procedure can successfully result in both audiovisual and lexical retuning effects, audiovisual cues, compared to lexical, can lead to larger effects (Ullas et al, 2020a; Figure 1).…”

“…In audiovisual processing, the visual cues such as lip movements are available earlier to the listener ( Jesse & Massaro, 2010), and thus strong perceptual shifts can be observed after only a few exposure items, but these effects also diminish quickly ( Vroomen, van Linden, Keetels, de Gelder, & Bertelson, 2004), whereas lexical cues can lead to longer-lasting, more robust effects, but after long exposures toward one particular phoneme (Eisner & McQueen, 2006). When lexical and audiovisual effects are compared under the same exposure and testing conditions, with short exposures (i.e., eight biasing items) in alternation with short categorization tests on ambiguous items, both adaptation effects occur, with audiovisual cues generating larger perceptual shifts than lexical cues (Ullas, Formisano, Eisner, & Cutler, 2020a;van Linden & Vroomen, 2007); the behavioral effects are however not additive (Ullas, Formisano, Eisner, & Cutler, 2020b).…”

“…The ambiguous phoneme blends, to be perceived differently depending on the prior exposure block, can consist of either edited words or videos. The exposure time can also be matched; although lexical retuning studies typically use longer exposure phases to induce a bias, such retuning can take place in shorter time spans and can be observed in shorter test blocks, similar to the typical audiovisual exposure, as well (Ullas et al, 2020a(Ullas et al, , 2020bvan Linden & Vroomen, 2007).…”

“…Ambiguous phonemes were presented in identical test blocks, and participants indicated their percept to assess recalibration effects in the same way for each exposure type. This approach of alternating exposure (containing either audiovisual or lexical stimuli, with changing phoneme biases) and test blocks has been shown to be effective in producing both audiovisual recalibration and lexical retuning (see Ullas et al, 2020a, for more details regarding the behavioral outcomes of this approach). By utilizing this procedure, the study aimed to identify the neural commonalities between lexical and audiovisual recalibration under similar experimental constraints, as well as potential unique contributions from multimodal or visual regions for audiovisual recalibration, in contrast to activity within areas of the language network for lexical retuning.…”