Cross-modal Association between Auditory and Visuospatial Information in Mandarin Tone Perception in Noise by Native and Non-native Perceivers

Abstract

Speech perception involves multiple input modalities. Research has indicated that perceivers establish cross-modal associations between auditory and visuospatial events to aid perception. Such intermodal relations can be particularly beneficial for speech development and learning, where infants and non-native perceivers need additional resources to acquire and process new sounds. This study examines how facial articulatory cues and co-speech hand gestures mimicking pitch contours in space affect non-native Mandarin tone perception. Native English as well as Mandarin perceivers identified tones embedded in noise with either congruent or incongruent Auditory-Facial (AF) and Auditory-FacialGestural (AFG) inputs. Native Mandarin results showed the expected ceiling-level performance in the congruent AF and AFG conditions. In the incongruent conditions, while AF identification was primarily auditory-based, AFG identification was partially based on gestures, demonstrating the use of gestures as valid cues in tone identification. The English perceivers’ performance was poor in the congruent AF condition, but improved significantly in AFG. While the incongruent AF identification showed some reliance on facial information, incongruent AFG identification relied more on gestural than auditory-facial information. These results indicate positive effects of facial and especially gestural input on non-native tone perception, suggesting that cross-modal (visuospatial) resources can be recruited to aid auditory perception when phonetic demands are high. The current findings may inform patterns of tone acquisition and development, suggesting how multi-modal speech enhancement principles may be applied to facilitate speech learning.

Degree of Language Experience Modulates Visual Attention to Visible Speech and Iconic Gestures During Clear and Degraded Speech Comprehension

“…As noise decreases the available acoustic information in the speech signal, it might be more difficult for non‐native listeners to make a phonological mapping between the speech signal and perceptual/linguistic representations, as these might have not been fully tuned to the non‐native language (Flege, ; Iverson et al, ; Lecumberri et al, ). Specifically in such situations, visual phonological information that is conveyed by visible speech has been shown to enhance non‐native language learning and comprehension (Hannah et al, ; Jongman, Wang, & Kim, ; Kawase, Hannah, & Wang, ; Kim, Sonic, & Davis, ; Wang, Behne, & Jiang, ). In native listeners, it has been suggested that visual attention is more often directed to the mouth of a talker to extract more information from visible speech when speech is degraded (Buchan, Paré, & Munhall, ; Król, ; Munhall, ; Rennig, Wegner‐Clemens, & Beauchamp, ).…”

Emblem Gestures Improve Perception and Evaluation of Non-native Speech

“…For example, Kelly et al (2009) investigated how semantic congruence of gesture and speech affected the learning of L2 Japanese vocabulary in native English speakers. Results from a free recall and recognition test showed that compared to speech alone, congruent gestures enhanced memory and incongruent gesture disrupted it (and see Hannah et al, 2017 , for a similar effect in L2 phonetic processing). Based on research in this vein, Macedonia (2014) makes a strong case for why hand gestures should be a bigger part of the L2 classroom and language education more generally.…”

“…Many of the experiments on this topic have focused on how L2 learners attend to information conveyed through the hands when perceiving novel speech sounds ( Hannah et al, 2017 ; Kelly, 2017 ; Kushch et al, 2018 ; Baills et al, 2019 ; Hoetjes et al, 2019 ) and comprehending new vocabulary ( Allen, 1995 ; Sueyoshi and Hardison, 2005 ; Sime, 2006 ; Kelly et al, 2009 ; Morett, 2014 ; Morett and Chang, 2015 ; Baills et al, 2019 ; Huang et al, 2019 ). For example, Kelly et al (2009) investigated how semantic congruence of gesture and speech affected the learning of L2 Japanese vocabulary in native English speakers.…”

“…These predictions were based on the following two lines of research as summarized in the introduction: one line of research showing that the relationship of gestures to speech matters for phonetic and semantic comprehension in L1 ( Popelka and Berger, 1971 ; McNeill et al, 1994 ; Kelly et al, 2010 ) and L2 ( Kelly et al, 2009 ; Hannah et al, 2017 ), and another line of research showing that the presence of meaningful gestures helps manage social impressions of L1 ( Maricchiolo et al, 2009 ) and L2 speakers ( Gullberg, 1998 ; Gregersen, 2005 ; McCafferty and Stam, 2009 ).…”

How Do Infants Disaggregate Referential and Affective Pitch?

“…By 4–6 months of age, infants in spite of their reduced visual processing can discriminate their native language from other languages partly by relying on visual cues accompanying gestures such as vocalic lip rounding (Weikum et al, 2007 ). In comparison, visual cues to tonal gestures are weak and unreliable to native listeners (Chen and Massaro, 2008 ; Hannah et al, 2017 ). Young infants (4-month-olds) can detect different emotions (happy, angry, sad) when presented with facial-vocal cues (Flom and Bahrick, 2007 ), an ability emerging prior to affect detection based on unimodal cues (Walker-Andrews, 1997 ).…”