The Kinematics of Social Action: Visual Signals Provide Cues for What Interlocutors Do in Conversation

Abstract

During natural conversation, people must quickly understand the meaning of what the other speaker is saying. This concerns not just the semantic content of an utterance, but also the social action (i.e., what the utterance is doing—requesting information, offering, evaluating, checking mutual understanding, etc.) that the utterance is performing. The multimodal nature of human language raises the question of whether visual signals may contribute to the rapid processing of such social actions. However, while previous research has shown that how we move reveals the intentions underlying instrumental actions, we do not know whether the intentions underlying fine-grained social actions in conversation are also revealed in our bodily movements. Using a corpus of dyadic conversations combined with manual annotation and motion tracking, we analyzed the kinematics of the torso, head, and hands during the asking of questions. Manual annotation categorized these questions into six more fine-grained social action types (i.e., request for information, other-initiated repair, understanding check, stance or sentiment, self-directed, active participation). We demonstrate, for the first time, that the kinematics of the torso, head and hands differ between some of these different social action categories based on a 900 ms time window that captures movements starting slightly prior to or within 600 ms after utterance onset. These results provide novel insights into the extent to which our intentions shape the way that we move, and provide new avenues for understanding how this phenomenon may facilitate the fast communication of meaning in conversational interaction, social action, and conversation.

Specific facial signals associate with categories of social actions conveyed through questions

“…A detailed description of the corpus collection, as well as the methods used for social actions transcriptions, facial signals annotations, and interrater reliabilities, can be found in Nota et al [33] and Trujillo and Holler [51]. The preregistration for this study is available on the As Predicted website https://aspredicted.org/6VZ_L2K.…”

“…This resulted in a raw agreement of 76% and k = 0.70, indicating substantial agreement (for more details, see Trujillo and Holler [51]). No annotations were made when there was insufficient facial signal data due to head movements preventing full visibility or due to occlusions.…”

“…Mdn = median, min = minimum, max = maximum, IQR = interquartile range, ms = milliseconds. a The total number of social actions differs slightly to Trujillo and Holler [51], due to the coding of four additional social actions at the moment of analysis.…”

Visual bodily signals as core devices for coordinating minds in interaction

“…In conversational context, torso movements were found to differ between categories of social actions, with questions expressing a stance or sentiment being accompanied by larger torso movements than requests for information, for example. This may be related to the act of distancing oneself from or showing agreement with certain attitudes [ 78 ].…”

Bridging a sensory gap between deaf and hearing people–A plea for a situated design approach to sensory augmentation

“…Causes of such sensory variations can range from the natural shapes of ears and curvature of the eyes to trauma resulting in a severed cochlear nerve. In humans, most sensory variations do not hinder interaction between them since spontaneously generated cues by individuals sufficiently support mutual understanding (Trujillo and Holler, 2021). However, when those cues are lacking or differ too much due to larger Where the definition of deafness covers a wide range of hearing-loss, we, for sake of clarity, focus on the societal division between people who are profoundly deaf, who only hear sound of dB or higher, and fully hearing people, who have a hearing threshold above dB ( ).…”