Gestures and Phases: The Dynamics of Speech-Hand Communication

Abstract

We investigated how a listener's perceived meaning of a spoken sentence is influenced by the relative timing between a speaker's speech and accompanying hand gestures. Participants viewed a computer-animated character who uttered the phrase, "Put the book there now." while executing a simple right-handed beat gesture whose location relative to the utterance was precisely controlled in a frame-by-frame fashion. The participant's task consisted of making a judgment about two related aspects of the actor's perceived speech: (a) Which word was emphasized? and (b) How clear was the emphasis? That is, did it make sense? The results revealed that the perceived emphasis was determined by the timing (phasing) of the speaker's hand gesture. Furthermore, the clarity of the perceived emphasis (i.e., meaningfulness) was influenced by the affordances in the immediate environment of the speaker. Discussion addresses the primacy of ostensive specification and gesture in communicative events, the dynamics of speech-hand coordination during both actual and virtual dialogue, and the role of environmental affordances in grounding informative communicative acts in the ecology of organism-environment dynamics. The question of the extent to which hand gestures (and other movements sometimes referred to as "body language") might provide a basis for human communication continues to challenge theories of language and language development. The fact that most of us move our hands in spontaneous gesture even if we cannot be seen, for example when talking to a blind person or to someone by telephone (Iverson & Goldin-Meadow, 1997), suggests that a speaker's accompanying hand

Multisensory Integration: The Case of a Time Window of Gesture–Speech Integration

“…It suggests that there are multiple positions in a sentence where gesture-speech integration can take place (for a similar notion for not semantically related gesture-speech integration, see Leonard & Cummins, 2011;Treffner et al, 2008). At the moment, we hypothesize that those positions are at the content words of the sentence, 7 but this needs further confirmation.…”

“…Previous gesture research has shown that timing of gesture and speech information plays a crucial role in their integration (Habets et al, 2011;Leonard & Cummins, 2011;Obermeier et al, 2011;Treffner et al, 2008). When gesture and speech are in close synchrony, their integration is seemingly automatic or obligatory (for more details on the notion of automaticity see below, but see also Kelly, Özyürek, & Maris, 2010).…”

“…This gesture phase contains the largest amount of (semantic) information, and speakers have the tendency to produce stroke and lexical affiliate together (Levelt, Richardson, & Laheij, 1985). To date, most research exploring the timing of gesture-speech integration used so-called beat gestures (e.g., Leonard & Cummins, 2011;Treffner, Peter, & Kleidon, 2008). These rapid flicks of the hand differ significantly from the iconic gestures used in the present paper in several ways.…”

What Iconic Gesture Fragments Reveal about Gesture–Speech Integration: When Synchrony Is Lost, Memory Can Help

“…In other words, he suggests that if gestures and speech are synchronous, they should be integrated in a rather automatic way. So far, however, there has been little empirical work on the effects of gesture-speech synchronization in comprehension (but see Treffner, Peter, & Kleidon, 2008).…”

Dynamics and Languaging: Toward an Ecology of Language

“…Best (1995) and Fowler (1986Fowler ( , 1996 have proposed that speech, like other ecological events, can be directly perceived (Gibson, 1966;Kelley, 1986). Other investigators have explored language as a "coordination device" (Clark, 1996) in which people talking entrain in various ways with their own actions or with those of interlocutors (e.g., Giles, Coupland, & Coupland, 1991;Shockley, Santana, & Fowler, 2003;Treffner & Peter, 2002) and in which nonverbal aspects of speaking and listening (e.g., manual gestures) affect linguistic interpretation (Treffner, Peter, & Kleidon, 2008). As Cowley notes in his contribution to this issue, Gibson (e.g., 1979Gibson (e.g., /1986 himself wrote a little about language, usually focusing on utterances as instances of indirect (mediated) perception, more or less like picture perception.…”