Cross-modal association between auditory and visual-spatial information in Mandarin tone perception

Abstract

Speech perception involves multiple input modalities. Research has indicated that perceivers may establish a cross-modal association between auditory and visual-spatial events to aid perception. Such intermodal relations can be particularly beneficial for non-native perceivers who need additional resources to process challenging new sounds. This study examines how co-speech hand gestures mimicking pitch contours in space affect non-native Mandarin tone perception. Native English as well as Mandarin perceivers identified tones with either congruent or incongruent auditory-facial and gestural (AF/G) input. Perceivers also identified congruent and incongruent auditory-facial (A/F) stimuli. Native Mandarin results showed the expected ceiling-level performance in the congruent A/F and AF/G conditions. In the incongruent conditions, while A/F identification was primarily auditory-based, AF/G identification was partially based on gestures, demonstrating the use of gestures as valid cues in tone identification. The English perceivers’ performance was poor in the congruent A/F condition, but improved significantly in AF/G. While the incongruent A/F identification showed some reliance on facial information, incongruent AF/G identification relied more on gestural than auditory-facial information. These results indicate positive effects of facial and especially gestural input on non-native tone perception, suggesting that cross-modal (visual-spatial) resources can be recruited to aid auditory perception when phonetic demands are high.

Metaphoric Gestures Facilitate Perception of Intonation More than Length in Auditory Judgments of Non-Native Phonemic Contrasts

“…This make sense given research on native language processing showing that people rely on visual information more when they struggle to process auditory information (Sumby & Pollack, 1954; and specifically for hand gesture, Drijvers & Özyürek, 2016;Obermeier, Dolk, & Gunter, 2012). Although the present study cannot definitely determine whether the second or third account is correct, the results still make a novel contribution to the literature: In contrast to recent findings showing that metaphoric length gestures do not help with learning vowel length distinctions (Hirata & Kelly, 2010;Hirata et al, 2014), we have shown that metaphoric intonation (pitch) gestures do help non-native speakers process phonemic intonational information in FL speech (see also, Hannah et al, 2016). This finding is significant because it suggests that regardless of whether one type of FL speech contrast is inherently harder to process than the other, gesture plays different roles within each one, suggesting that some possible boundaries of gesture-speech integration at the phonemic level (more on this below).…”

“…The present findings are interesting in light of the recent study by Hannah et al (2016) also investigating the role of metaphoric hand gestures on English speakers processing of intonational contrasts (in Mandarin, not Japanese). Recall that in that study, participants were asked to attend to both speech and metaphoric gestures in order to judge what Mandarin tone they had just perceived.…”

“…Second, we build on previous work by using a more direct measure of gesture-speech integration than the paradigm used by Hannah et al (2016). In that study, judgments were made about tones without specifying whether participants should attend to speech or gesture.…”

“…First, it directly compared how metaphoric gestures affect perception of phonemic length (Hirata & Kelly, 2010) vs. intonational contrasts (Hannah et al, 2016). To accomplish this, we exposed English speakers to videos of a native Japanese speaker producing congruent and incongruent metaphoric gestures with Japanese sentences involving two types of acoustic distinctions, vowel length and intonational contour.…”

“…In a recent experiment by Hannah, Wang and colleagues (Hannah, Wang, Jongman, & Sereno, 2016), native-English speakers were exposed to Mandarin tones with congruent (rising hand and rising tone) and incongruent (rising hand and falling tone) metaphoric gestures.…”