Multi-Modal Perception of Tone

Wang, Yue; Sereno, Joan A.; Jongman, Allard

doi:10.1007/978-981-15-7606-5_9

Cited by 4 publications

(5 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, interactions of speech style and input modality would tease apart contributions of visual and auditory information in intelligibility gain, as well as identifying whether clear relative to plain tones are more visually distinctive. Understanding the extent to which clear speech benefits visual tone perception is particularly critical in unraveling the signal- versus code-based clear speech principles, given that tone is believed to be less visually distinct ( Wang et al, 2020 ). Thus, in what follows, the results are discussed in terms of the effects of visual clear tones, A/V modalities, and L1 backgrounds.…”

Section: Discussionmentioning

confidence: 99%

“…These results suggest that specific movements of the head, eyebrows and lips are correlated with tonal articulation, and are likely coordinated with the spatial and temporal dynamics of the production of different tones (see Wang et al, 2020 , for a recent review). However, further evidence from tone perception research is needed to determine if these facial tonal cues are linguistically relevant cues to enhance tone category distinctions in perception.…”

Section: Introductionmentioning

confidence: 91%

See 1 more Smart Citation

Multi-modal cross-linguistic perception of Mandarin tones in clear speech

Zeng,

Leung,

Jongman

et al. 2023

Front. Hum. Neurosci.

Self Cite

View full text Add to dashboard Cite

Clearly enunciated speech (relative to conversational, plain speech) involves articulatory and acoustic modifications that enhance auditory–visual (AV) segmental intelligibility. However, little research has explored clear-speech effects on the perception of suprasegmental properties such as lexical tone, particularly involving visual (facial) perception. Since tone production does not primarily rely on vocal tract configurations, tones may be less visually distinctive. Questions thus arise as to whether clear speech can enhance visual tone intelligibility, and if so, whether any intelligibility gain can be attributable to tone-specific category-enhancing (code-based) clear-speech cues or tone-general saliency-enhancing (signal-based) cues. The present study addresses these questions by examining the identification of clear and plain Mandarin tones with visual-only, auditory-only, and AV input modalities by native (Mandarin) and nonnative (English) perceivers. Results show that code-based visual and acoustic clear tone modifications, although limited, affect both native and nonnative intelligibility, with category-enhancing cues increasing intelligibility and category-blurring cues decreasing intelligibility. In contrast, signal-based cues, which are extensively available, do not benefit native intelligibility, although they contribute to nonnative intelligibility gain. These findings demonstrate that linguistically relevant visual tonal cues are existent. In clear speech, such tone category-enhancing cues are incorporated with saliency-enhancing cues across AV modalities for intelligibility improvements.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 91%

Multi-modal cross-linguistic perception of Mandarin tones in clear speech

Zeng,

Leung,

Jongman

et al. 2023

Front. Hum. Neurosci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Due to the vast array of visual cues involved in audiovisual speech perception, many researchers have attempted to categorise them and evaluate their specific roles. First, these cues can be categorised by their location (i.e., head, face, and neck) as was the case for Wang et al ( 2020 ). Second, from an acquisition perspective, Lalonde and Werner ( 2021 ) reviewed the audiovisual speech cues used by infants and children and highlighted the salience of temporal and phonetic information.…”

Section: Introductionmentioning

confidence: 99%

“…Preliminary evidence (discrimination task: Burnham et al, 2001 , 2015 , 2022 ; identification task: Mixdorff et al, 2005 ; Burnham et al, 2006 ; Chen and Massaro, 2008 ) suggests that visual benefit occurs in lexical tone perception also, similar to that enhancing the discrimination and identification of consonants and vowels. Wang et al ( 2020 ) categorised these visual cues into three types according to their origins, namely, head, eyebrows, and lip movements. However, as lexical tones are articulated by vibrations of the vocal cords, there are no apparent visemic cues from a speaker's mouth area.…”

Section: Introductionmentioning

confidence: 99%

“…In terms of lexical tone perception, many studies have revealed that non-native Chinese speakers, such as English or Dutch speakers, adopt universal perception strategies that allow them to utilise some visual cues (Burnham et al, 2015 ; Han et al, 2019 ). Wang et al ( 2020 ) noted that native tonal language speakers outperformed non-native tonal language speakers in an audiovisual condition; however, the non-natives showed superior performance in some cases of the visual-only condition (Smith and Burnham, 2012 ; Burnham et al, 2015 ). Burnham et al suggested that non-tonal language speakers explored facial cues as they found themselves in a challenging phonetic situation.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Incongruent visual cues affect the perception of Mandarin vowel but not tone

Hong

Wang²,

Zeng³

2023

Front. Psychol.

View full text Add to dashboard Cite

Over the recent few decades, a large number of audiovisual speech studies have been focusing on the visual cues of consonants and vowels but neglecting those relating to lexical tones. In this study, we investigate whether incongruent audiovisual information interfered with the perception of lexical tones. We found that, for both Chinese and English speakers, incongruence between auditory and visemic mouth shape (i.e., visual form information) significantly interfered with reaction time and reduced the identification accuracy of vowels. However, incongruent lip movements (i.e., visual timing information) did not interfere with the perception of auditory lexical tone. We conclude that, in contrast to vowel perception, auditory tone perception seems relatively impervious to visual congruence cues, at least under these restricted laboratory conditions. The salience of visual form and timing information is discussed based on this finding.

show abstract