Language Experience Changes Audiovisual Perception

Abstract

Can experience change perception? Here, we examine whether language experience shapes the way individuals process auditory and visual information. We used the McGurk effect—the discovery that when people hear a speech sound (e.g., “ba”) and see a conflicting lip movement (e.g., “ga”), they recognize it as a completely new sound (e.g., “da”). This finding that the brain fuses input across auditory and visual modalities demonstrates that what we hear is profoundly influenced by what we see. We find that cross-modal integration is affected by language background, with bilinguals experiencing the McGurk effect more than monolinguals. This increased reliance on the visual channel is not due to decreased language proficiency, as the effect was observed even among highly proficient bilinguals. Instead, we propose that the challenges of learning and monitoring multiple languages have lasting consequences for how individuals process auditory and visual information.

Bridging ears and eyes when learning spoken words: On the effects of bilingual experience at 30 months

“…Moving beyond lexical considerations alone, it could be the case that bilingualism hones low‐level cross‐modal exchanges between the auditory and visual modality. This is supported by brain imagery evidence showing that bilingual children (Della Rosa et al., 2013) and adults (Li, Legault, & Litcofsky, 2014) have more gray matter density and cortical thickness in two multisensory pathways that are highly engaged in audio–visual speech processing and are more inclined to bind auditory and visual features of speech (McGurk illusion, Marian, Hayakawa, Lam, & Schroeder, 2018) and non‐speech events (Double flash illusion, Bidelman & Heath, 2019. Sound‐color synesthesia, Ward, 2013; Watson et al., 2017).…”

The Case for Measuring and Reporting Bilingualism in Developmental Research

“…Bilingual adolescents and adults also show different patterns of speech perception and encoding, even for low-level information such as the fundamental frequency of speech syllables (Krizman, Marian, Shook, Skoe, & Kraus, 2012). Integration of auditory and visual information is affected by bilingualism: bilinguals are less susceptible to illusions that fuse asynchronous nonlinguistic auditory and visual stimuli into a single percept (Bidelman & Heath, 2018), but are more susceptible to such illusions with mismatching audiovisual speech syllables (Marian, Hayakawa, Lam, & Schroeder, 2018). There are also effects of linguistic and cultural immersion for how adults perceive and process color, even in pre-attentive tasks (Athanasopoulos, Dering, Wiggett, Kuipers, & Thierry, 2010), as well as for how speakers of different languages process the visual world and perform in visual search tasks (Chabal, Schroder, & Marian, 2015).…”

Neural differences in the temporal cascade of reactive and proactive control for bilinguals and monolinguals

“…Since past research has shown differences in language processing between balanced and unbalanced bilinguals (e.g., Rosselli et al, 2016), it was important to recruit a sample with balanced bilinguals for the present study. Therefore, bilinguals' language abilities were evaluated using self-report data from the Language Experience and Proficiency Questionnaire (LEAP-Q; Marian et al, 2007). This scale asked them to rate their understanding of their languages, as well as their speaking and reading skills in both languages, on a scale from 0 to 10.…”