Neural Correlates of Phonetic Adaptation as Induced by Lexical and Audiovisual Context

Abstract

When speech perception is difficult, one way listeners adjust is by reconfiguring phoneme category boundaries, drawing on contextual information. Both lexical knowledge and lipreading cues are used in this way, but it remains unknown whether these two differing forms of perceptual learning are similar at a neural level. This study compared phoneme boundary adjustments driven by lexical or audiovisual cues, using ultra-high-field 7-T fMRI. During imaging, participants heard exposure stimuli and test stimuli. Exposure stimuli for lexical retuning were audio recordings of words, and those for audiovisual recalibration were audio–video recordings of lip movements during utterances of pseudowords. Test stimuli were ambiguous phonetic strings presented without context, and listeners reported what phoneme they heard. Reports reflected phoneme biases in preceding exposure blocks (e.g., more reported /p/ after /p/-biased exposure). Analysis of corresponding brain responses indicated that both forms of cue use were associated with a network of activity across the temporal cortex, plus parietal, insula, and motor areas. Audiovisual recalibration also elicited significant occipital cortex activity despite the lack of visual stimuli. Activity levels in several ROIs also covaried with strength of audiovisual recalibration, with greater activity accompanying larger recalibration shifts. Similar activation patterns appeared for lexical retuning, but here, no significant ROIs were identified. Audiovisual and lexical forms of perceptual learning thus induce largely similar brain response patterns. However, audiovisual recalibration involves additional visual cortex contributions, suggesting that previously acquired visual information (on lip movements) is retrieved and deployed to disambiguate auditory perception.

Cortical responses to letters and ambiguous speech vary with reading skills in dyslexic and typically reading children

“…Previous research has shown that the MRI environment reduces attentional focus on the task ( van Maanen et al, 2016 ) and 8–10 year old children who are still developing their attentional skills ( Amso and Scerif, 2015 , Betts et al, 2006 , Klenberg et al, 2001 , Klimkeit et al, 2004 , Lin et al, 1999 ) may be more prone to such effects. Although somewhat reduced compared to offline behavioural experiments, in adults behavioural recalibration effects tend to be preserved in the MRI setting ( Bonte et al, 2017 , Kilian-Hütten et al, 2011a , Ullas et al, 2020b ). Thus, a developmental trajectory of both the text-based recalibration effect and more general cognitive and attentional mechanisms may underlie the differences in text-based recalibration performance in the MRI environment between children and adults.…”

Adaptive Plasticity in Perceiving Speech Sounds

“…In contrast, some areas of the brain may be uniquely engaged by either recalibration or retuning. When compared directly using fMRI within the same participants, audiovisual recalibration and lexical retuning showed largely similar areas of activation, over temporal, parietal, and motor cortex areas, although audiovisual recalibration specifically seems to retrigger activation within areas of the visual cortex, despite the lack of visual stimuli during the recalibration test trials (Ullas et al 2020).…”

Visuospatial attention revamps cortical processing of sound: restrict stimulus uncertainty

“…Proximity is in such cases indexed by temporal correlation, a variable that underscores the integration of audiovisual (AV) human communication signals at perceptual and neural levels (Spence & Squire, 2003;Holmes & Spence, 2005;Roseboom et al, 2009;Park et al, 2016). For example, unimodal processing in joint visual face and speech sounds can be routinely biased according to coherent, unified percepts that emerge to disambiguate single unimodal contexts and disappear as sensory streams become desynchronyzed (Andersen et al, 2009;McGurk & MacDonald, 1976;Miller & D'Esposito, 2005;Ullas et al, 2020;van Atteveldt et al, 2007).…”