Co-Speech Movement in Conversational Turn-Taking

Abstract

This study investigates co-speech movements as a function of the conversational turn exchange type, the type of speech material at a turn exchange, and the interlocutor’s role as speaker or listener. A novel interactive protocol that mixes conversation and (non-read) nursery rhymes works to elicit many speech turns and co-speech movements within dyadic speech interaction. To evaluate a large amount of data, we use the density of co-speech movement as a quantitative measure. Results indicate that both turn exchange type and participant role are associated with variation in movement density for head and brow co-speech movement. Brow and head movement becomes denser as speakers approach overlapping speech exchanges, indicating that speakers increase their movement density as an interruptive exchange is approached. Similarly, head movement generally increases after such overlapping exchanges. Lastly, listeners display a higher rate of co-speech movement than speakers, both at speech turns and remote from them. Brow and head movements generally behave similarly across speech material types, conversational roles, and turn exchange types. On the whole, the study demonstrates that the quantitative co-speech movement density measure advanced here is useful in the study of co-speech movement and turn-taking.

The Temporal Alignment of Speech-Accompanying Eyebrow Movement and Voice Pitch: A Study Based on Late Night Show Interviews

“…Flecha-García [55,56] found that brows tend to be raised in 'instruct moves' in dialogue games used for elicitation (the 'Map task' [57,58], e.g., "So then you go along just a few dashes" [55] (p. 68)). The role of eyebrow movement in turn-taking has been noticed in several studies [55,[59][60][61], e.g., insofar as, "more frequent and longer eyebrow-raising occurred in the initial utterance of high-level discourse segments than anywhere else in the dialogue" [55] (p. ii).…”

Rhythm is a timescale

“…We make a distinction between speech in spontaneous communication scenarios and speech types that incorporate musical aspects, which may include prayers, mantras, chants, poems, nursery rhymes, freestyle rap and many other types of belief-based and/or artistic expressions (see Leong & Goswami 2015, Fuchs & Reichel 2016, Davis 2017, Cummins 2018, Danner et al 2021. All the latter tend to incorporate quasi-isochrony which should be related to the musical aspect of these complex fusions between music and speech.…”

Between Reality and Delusion: Challenges of Applying Large Language Models to Companion Robots for Open-Domain Dialogues with Older Adults

“…While gaze aversion and light indication helped decrease the interruptions, but they still occurred in most of the conversations. Turn-taking behavior can be further improved by using dialogue context (e.g., [75]), speech prosody (e.g., [76]), estimating the user's gaze, gestures, and facial expressions, such as eyebrow movement and mouth opening [77], or a combination of these features, e.g., [78][79][80] (for an in-depth review of turn-taking in HRI and conversational systems, see [81]) to prevent interrupting the user, both for improving user experience and speech recognition.…”