Developmental shifts in children’s sensitivity to visual speech: A new multimodal picture–word task

Abstract

This research developed a Multimodal Picture Word Task for assessing the influence of visual speech on phonological processing by100 children between 4 -14 yrs of age. We assessed how manipulation of seemingly to-be-ignored auditory (A) and audiovisual (AV) phonological distractors affected picture naming without participants consciously trying to respond to the manipulation. Results varied in complex ways as a function of age and type and modality of distractors. Results for congruent AV distractors yielded an inverted U-shaped function with a significant influence of visual speech in 4-yr-olds and 10-14-yr-olds, but not in 5-9-yr-olds. In concert with dynamic systems theory, we proposed that the temporary loss of sensitivity to visual speech was reflecting reorganization of relevant knowledge and processing sub-systems, particularly phonology. We speculated that reorganization may be associated with 1) formal literacy instruction and 2) developmental changes in multimodal processing and auditory perceptual, linguistic, and cognitive skills. KeywordsPicture-Word Task; Audiovisual Speech Perception; U-Shaped Developmental Function; Phonological Processing; Picture-Word Interference; Picture Naming; Multimodal Speech Processing; Dynamic Systems Theory Speech communication by adults is naturally a multimodal event with auditory and visual speech integrated mandatorily. This basic property of mature speech perception is dramatically illustrated by McGurk effects (McGurk & McDonald, 1976). In the McGurk task, individuals hear a syllable whose onset has one place of articulation while seeing a talker simultaneously mouthing a syllable whose onset has a different place of articulation, e.g., auditory /ba/ and visual /ga/. Adults typically experience the illusion of perceiving /da/ or /ða/, a blend of the auditory and visual inputs. The McGurk illusion is consistent with the idea that auditory and visual speech interact prior to the classification of phonetic features, such as place of Address correspondence to: Susan Jerger, Ph.D., School of Behavioral and Brain Sciences, University of Texas at Dallas, P. O. Box 830688, GR4.1, Richardson, TX 75083-0688. Phone -972-883-2268; FAX -972-883-2491; sjerger@utdallas.edu. Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. (Green, 1998). Integrating auditory and visual speech without conscious effort clearly has adaptive value. Seeing a talker's face facilitates listening in noisy soundscapes and in clear environments containing unfamiliar or complex content (Arnold & Hill, 2001;MacLeod & Summerfield, 1987;Massaro, 1998). NIH Publ...

Preschoolers Benefit From Visually Salient Speech Cues

“…Although children 3 years and older evidence AV benefit (Dodd, 1977;Holt et al, 2011;Ross et al, 2011;Wightman et al, 2006), younger children benefit less than adults (Ross et al, 2011). And in some cases, young children show no AV benefit whatsoever (Jerger, Damian, Spence, Tye-Murray, & Abdi, 2009;Wightman et al, 2006). In addition, whereas adults typically fuse discrepant auditory and visual (McGurk) stimuli or report the visual percept, children are much less likely to fuse the auditory and visual stimuli and typically report the auditory percept (Desjardins, Rogers, & Werker, 1997;Massaro, 1984;Massaro, Thompson, Barron, & Laren, 1986;McGurk & MacDonald, 1976).…”

“…Mixed findings regarding AV integration in young children have led some to posit that AV speech integration development follows a U-shaped trajectory (Jerger et al, 2009), which has been explained using dynamic systems theory (Smith & Thelen, 2003). According to this theory, development does not proceed monotonically; rather, there are periods of disorganization and behavioral regression associated with the emergence of new states of organization (Gershkoff-Stowe & Thelen, 2004;Markovitch & Lewkowicz, 2004).…”

“…From this perspective, the plateau of the function (when visual speech lacks influence) reflects a period of transition rather than a "loss" of skill. While phonetic representations reorganize, visual phonetic information is thought to be harder to access (Jerger et al, 2009). Two periods of development are commonly associated with phonological reorganization: (a) the vocabulary growth spurt that begins around 18 months of age and extends into middle childhood (Walley, Metsala, & Garlock, 2003) and (b) early literacy training that extends between 6 and 9 years of age (Anthony & Francis, 2005;Morais, Bertelson, Cary, & Algeria, 1986).…”

“…Two periods of development are commonly associated with phonological reorganization: (a) the vocabulary growth spurt that begins around 18 months of age and extends into middle childhood (Walley, Metsala, & Garlock, 2003) and (b) early literacy training that extends between 6 and 9 years of age (Anthony & Francis, 2005;Morais, Bertelson, Cary, & Algeria, 1986). Most importantly, dynamic systems theory posits that multiple factors-including perceptual skills, phonological representations/knowledge, and general attention resources (Jerger et al, 2009)-interact to account for the U-shaped trajectory. This study emphasizes one factor that might play an important role in the mixed findings regarding visual speech influence in young children (and thus the U-shaped developmental trajectory): differences in the non-sensory cognitive and linguistic (phonological) requirements of tasks used to assess children's AV integration across studies.…”

“…Children are required to make overt responses, including verbal responses (Boothroyd, Eisenberg, & Martinez, 2010;Dodd, 1977;Holt et al, 2011;Jerger et al, 2009;Massaro et al, 1986;McGurk & MacDonald, 1976;Ross et al, 2011;Tremblay et al, 2007) and other motor responses (Desjardins et al, 1997;Hnath-Chisolm, Laipply, & Boothroyd, 1998;Massaro et al, 1986;Sekiyama & Burnham, 2008;Wightman et al, 2006). Because the same tasks are used to assess children and adults, many of the developmental results-in particular, findings of reduced visual influence and AV advantage in young children on some measures-might reflect differences in non-sensory processing efficiency and susceptibility to task demands, rather than age-related changes in AV integration (Allen & Wightman, 1992;Boothroyd, 1991;Wightman, Allen, Dolan, Kistler, & Jamieson, 1989).…”