Gesture Influences Resolution of Ambiguous Statements of Neutral and Moral Preferences

Abstract

When faced with an ambiguous pronoun, comprehenders use both multimodal cues (e.g., gestures) and linguistic cues to identify the antecedent. While research has shown that gestures facilitate language comprehension, improve reference tracking, and influence the interpretation of ambiguous pronouns, literature on reference resolution suggests that a wide set of linguistic constraints influences the successful resolution of ambiguous pronouns and that linguistic cues are more powerful than some multimodal cues. To address the outstanding question of the importance of gesture as a cue in reference resolution relative to cues in the speech signal, we have previously investigated the comprehension of contrastive gestures that indexed abstract referents – in this case expressions of personal preference – and found that such gestures did facilitate the resolution of ambiguous statements of preference. In this study, we extend this work to investigate whether the effect of gesture on resolution is diminished when the gesture indexes a statement that is less likely to be interpreted as the correct referent. Participants watched videos in which a speaker contrasted two ideas that were either neutral (e.g., whether to take the train to a ballgame or drive) or moral (e.g., human cloning is (un)acceptable). A gesture to the left or right side co-occurred with speech expressing each position. In gesture-disambiguating trials, an ambiguous phrase (e.g., I agree with that, where that is ambiguous) was accompanied by a gesture to one side or the other. In gesture non-disambiguating trials, no third gesture occurred with the ambiguous phrase. Participants were more likely to choose the idea accompanied by gesture as the stimulus speaker’s preference. We found no effect of scenario type. Regardless of whether the linguistic cue expressed a view that was morally charged or neutral, observers used gesture to understand the speaker’s opinion. This finding contributes to our understanding of the strength and range of cues, both linguistic and multimodal, that listeners use to resolve ambiguous references.

A multimodal approach to polysemy: the senses of touch

“…They found that when speakers are asked to clarify the meaning of an ambiguous word used in the course of an extended discourse (in their work, homonyms), they are more likely to use a gesture in association with the problematic word; similar results were found by Holle and Gunter (2007). Similarly, when investigating the resolution of ambiguous statements related to neutral and moral preferences, Hinnell and Parrill (2020) showed that participants rely on gesture, when available, to understand the speaker's ambiguous statements. Particularly, gesture appeared to be used by observers to understand the speaker's opinion, since they were more likely to choose the idea accompanied by a gesture as the speaker's preference, regardless of whether the linguistic cue expressed a neutral idea or a moral view that might be considered socially unacceptable.…”

Embodying Similarity and Difference: The Effect of Listing and Contrasting Gestures During U.S. Political Speech

“…More recently, scholars started to consider the ways in which gesture can impact the perception of abstract entities, such as arguments or conceptual ideas (Hinnell & Parrill, 2020; Parrill & Stec, 2017; Parrill et al., 2022). For example, Hinnell and Parrill (2020) presented participants with videos of speakers contrasting two ideas each accompanied by a gesture to the right or the left (e.g., “My little brother's not on Facebook because he thinks it's a waste of time” + gesture to right “but my other brother says he can't do job networking without it” + gesture to left). The speakers then expressed a preference or alignment with one of the ideas using an ambiguous pronoun (e.g., “I agree with him”).…”

“…However, many past studies have focused on how gestural input impacts addressees’ perception of concrete entities (e.g., where characters in a story were or how they acted), which already have physical spatial features that can map onto gesture. More recently, scholars started to consider the ways in which gesture can impact the perception of abstract entities, such as arguments or conceptual ideas (Hinnell & Parrill, 2020; Parrill & Stec, 2017; Parrill et al., 2022). For example, Hinnell and Parrill (2020) presented participants with videos of speakers contrasting two ideas each accompanied by a gesture to the right or the left (e.g., “My little brother's not on Facebook because he thinks it's a waste of time” + gesture to right “but my other brother says he can't do job networking without it” + gesture to left).…”

Gesture links language and cognition for spoken and signed languages

“…Co-speech gesture expressing the meaning of novel words can help children learn the words [170][171][172][173][174][175][176][177][178] . Furthermore, if gestures associate multiple referents at multiple locations, a subsequent gesture at one of the locations can disambiguate an ambiguous expression in speech [179][180][181] (also true for deictic co-sign gesture that accompanies pronouns 182 ).…”