Perception of the multisensory coherence of fluent audiovisual speech in infancy: Its emergence and the role of experience

Abstract

To investigate the developmental emergence of the ability to perceive the multisensory coherence of native and non-native audiovisual fluent speech, we tested 4-, 8–10, and 12–14 month-old English-learning infants. Infants first viewed two identical female faces articulating two different monologues in silence and then in the presence of an audible monologue that matched the visible articulations of one of the faces. Neither the 4-month-old nor the 8–10 month-old infants exhibited audio-visual matching in that neither group exhibited greater looking at the matching monologue. In contrast, the 12–14 month-old infants exhibited matching and, consistent with the emergence of perceptual expertise for the native language, they perceived the multisensory coherence of native-language monologues earlier in the test trials than of non-native language monologues. Moreover, the matching of native audible and visible speech streams observed in the 12–14 month olds did not depend on audio-visual synchrony whereas the matching of non-native audible and visible speech streams did depend on synchrony. Overall, the current findings indicate that the perception of the multisensory coherence of fluent audiovisual speech emerges late in infancy, that audio-visual synchrony cues are more important in the perception of the multisensory coherence of non-native than native audiovisual speech, and that the emergence of this skill most likely is affected by perceptual narrowing.

Audio-Visual Perception of Gender by Infants Emerges Earlier for Adult-Directed Speech

“…Even though infants are from birth able to match individual voice characteristics with particular faces [70, 71], and perceive multisensory coherence of visible and audible speech syllables from 2 to 4 months of age [17–19], our results provide further evidence that it is not until the middle of the first year of life that they acquire the necessary skills to perceive more complex attributes such as gender. It may be that increased experience is needed before multisensory coherence emerges for such attributes [20, 24, 72]. …”

Developmental Change in Infants' Detection of Visual Faces that Match Auditory Vowels

“…For example, conflicting gender information can sometimes interfere with 4-month-olds' AV matching (Patterson & Werker, 2002), and Altvater- Mackensen and Grossmann (2015) show that AV matching performance is variable with more difficult vowels, and performance is predicted by aspects of mothers' speech input and infants' vocal productivity. In a similar vein, Lewkowicz and colleagues found that infants have difficulty matching fluent passages of auditory speech to talking faces, but by 12-14 months of age, infants succeed at AV matching for these more complex stimuli, especially when in the native language (Lewkowicz, Minar, Tift, & Brandon, 2015;Lewkowicz & Pons, 2013). In a similar vein, Lewkowicz and colleagues found that infants have difficulty matching fluent passages of auditory speech to talking faces, but by 12-14 months of age, infants succeed at AV matching for these more complex stimuli, especially when in the native language (Lewkowicz, Minar, Tift, & Brandon, 2015;Lewkowicz & Pons, 2013).…”

Multisensory Processes: A Balancing Act across the Lifespan

“…By five months, infants become susceptible to the multisensory speech illusion known as the McGurk effect [37], which indexes the binding of auditory and visual speech elements. By 6–9 months of age infants begin to detect gender [38] and affect [39] as bound multisensory perceptual constructs, by 8 months they start to selectively attend to the source of fluent audiovisual speech located in a talker’s mouth [2], and by 12 months they begin to perceive the multisensory coherence of fluent audiovisual speech and the multisensory identity of their native speech [40,41] (Figure 2B). This general developmental picture is the same for pairings beyond vision and audition (e.g.…”