Using the Hands to Identify Who Does What to Whom: Gesture and Speech Go Hand‐in‐Hand

Abstract

In order to produce a coherent narrative, speakers must identify the characters in the tale so that listeners can figure out who is doing what to whom. This paper explores whether speakers use gesture, as well as speech, for this purpose. English speakers were shown vignettes of two stories and asked to retell the stories to an experimenter. Their speech and gestures were transcribed and coded for referent identification. A gesture was considered to identify a referent if it was produced in the same location as the previous gesture for that referent. We found that speakers frequently used gesture location to identify referents. Interestingly, however, they used gesture most often to identify referents that were also uniquely specified in speech. Lexical specificity in referential expressions in speech thus appears to go hand-in-hand with specification in referential expressions in gesture.

The role of intonation and visual cues in the perception of sentence types: Evidence from European Portuguese varieties

“…This finding also points to the strength of intonation over visual cues. Additionally, we concluded that SEP participants do not necessarily need more time to react when being exposed to incongruent stimuli, as suggested by previous studies such as Chen (2003), and Schneider et al (2011): Reaction times of SEP participants were shorter when exposed to manipulated stimuli from their native variety, than when exposed to original stimuli.…”

“…As mentioned in Section 2.3, reaction times (RTs) were also measured in order to observe whether participants need more time to react when being exposed to stimuli that are more difficult to categorize (Chen, 2003;Schneider et al, 2011). In this sense, we expect longer reaction times in the VO condition than in the AO and AV conditions for both varieties (Hypothesis 5).…”

“…Overall, participants of each variety had to decide about the sentence type of a total of 84 trials (24 AO, 24 VO, 12 original AV from SEP, 12 original AV from PtD, 12 manipulated AV from SEP) and a total of 1680 responses were obtained (84 trials × 20 participants) for each variety. Reaction times (in milliseconds) were also measured in order to observe whether participants need more time to react when being exposed to stimuli that are more difficult to categorize (Chen, 2003;Schneider et al, 2011). Thus, we expect longer reaction times in the VO condition than in the AO and AV conditions (Hypothesis 5).…”

“…Some studies point to the hand-in-hand hypothesis (So et al, 2009;de Ruiter et al, 2012), assuming that gestures are redundant in the sense that they basically express information that can reliably be derived from the verbal content alone. Other studies propose an alternative hypothesis, based on the assumption of a trade-off relation between gestures and speech production, i.e., speech and gestures complement each other (Bangerter, 2004;Melinger & Levelt, 2004).…”

Development of multimodal discourse comprehension: cohesive use of space by gestures

“…For example, when introducing a new protagonist in a narrative, adult speakers often assign them to a specific area by placing or directing a gesture to the area. When the same referent is mentioned again later, the same location is gesturally indicated (Gullberg, 2006;So, Kita, & Goldin-Meadow, 2009). In other words, once a location is assigned to a particular referent, it is often maintained throughout the discourse, not unlike the use of space for co-reference in sign language (e.g., Bellugi & Klima, 1982).…”