Beat gestures influence which speech sounds you hear

Abstract

Beat gestures – spontaneously produced biphasic movements of the hand – are among the most frequently encountered co-speech gestures in human communication. They are closely temporally aligned to the prosodic characteristics of the speech signal, typically occurring on lexically stressed syllables. Despite their prevalence across speakers of the world’s languages, how beat gestures impact spoken word recognition is unclear. Can these simple ‘flicks of the hand’ influence speech perception? Across six experiments, we demonstrate that beat gestures influence the explicit and implicit perception of lexical stress (e.g., distinguishing OBject from obJECT), and in turn, can influence what vowels listeners hear. Thus, we provide converging evidence for a manual McGurk effect: even the simplest ‘flicks of the hands’ influence which speech sounds we hear.SIGNIFICANCE STATEMENTBeat gestures are very common in human face-to-face communication. Yet we know little about their behavioral consequences for spoken language comprehension. We demonstrate that beat gestures influence the explicit and implicit perception of lexical stress, and, in turn, can even shape what vowels we think we hear. This demonstration of a manual McGurk effect provides some of the first empirical support for a recent multimodal, situated psycholinguistic framework of human communication, while challenging current models of spoken word recognition that do not yet incorporate multimodal prosody. Moreover, it has the potential to enrich human-computer interaction and improve multimodal speech recognition systems.

Beat Gestures for Comprehension and Recall: Differential Effects of Language Learners and Native Listeners

“…Thus it is possible that differences in the phonetic realization of prominence may have had an effect. Conversely, other studies have also shown that when prosody is held constant, the presence of a beat gesture boosts the perception of speech prominence (Krahmer and Swerts, 2007;Bosker and Peeters, 2020). Even though our materials were controlled for the presence of pitch accentuation in beat positions across conditions, the fact that speech production was not kept completely constant does not rule out the possibility that pitch range differences might have had an effect on the results.…”

“…Indeed, non-referential beat gestures are one of the most common types of gesture that are produced by speakers, particularly the case in academic contexts where these gestures predominate at rates of up to 94.6% of the gesture types produced (Shattuck-Hufnagel and Ren, 2018, see also Rohrer et al, 2019 for similar results). These gestures (much like their referential counterparts) are also integrated with speech prosody (often co-occurring with pitch accentuation), and their presence can actually modulate a listener's perception of prominence (see Krahmer and Swerts, 2007;Bosker and Peeters, 2020). Further, non-referential beat gestures have important discursive and pragmatic functions, such as marking information structure (Im and Baumann, 2020), epistemic stance (Prieto et al, 2018;Shattuck-Hufnagel and Prieto, 2019), among others.…”

The importance of visual control and biomechanics in the regulation of gesture-speech synchrony for an individual deprived of proprioceptive feedback of body position

“…It is possible that IW's gestures might not lose their efficacy when more variably timed; i.e., a statistical change then is not communicatively meaningful. On the other hand, gesture-speech timing incoherencies are in line with IW's phenomenology of a perceived sense of effort to gesture under the blind [5] , and it has been found that humans are able to be perceptually sensitive to syllable-level timing differences in gesture [44], [45]; in this sense the current timing variability is likely a meaningful difference.…”

“…While difference in timing variability between the blind and non-blind conditions is statistically clear, whether these differences impact communication is an empirical issue [44], [45]. It is possible that IW's gestures might not lose their efficacy when more variably timed; i.e., a statistical change then is not communicatively meaningful.…”

Kaleidoscope

“…At the micro-level, a paper by Bosker & Peeters re-examines the ‘McGurk effect’ – where an auditory stimulus ‘ba’ is accompanied by a visual stimulus of the speaker's mouth movements that articulate a different sound ‘ga’ resulting in participants consistently reporting hearing ‘da’. 5 In the McGurk experiment, the visual stimulus strongly biases the auditory percept. In Bosker & Peeters’ study, instead of using video of the speaker's mouth movement, they examine how ‘beat gestures’ influence speech perception.…”