Conversational eyebrow frowns facilitate question identification: An online VR study

Abstract

Conversation is a time-pressured environment. Recognising a social action (the ‘speech act’, such as a question requesting information) early is crucial in conversation to quickly understand the intended message and plan a timely response. Fast turns between interlocutors are especially relevant for responses to questions, since a long gap may be meaningful by itself. Human language is multimodal, involving speech as well as visual signals from the body, including the face. But little is known about how conversational facial signals contribute to the communication of social actions. Some of the most prominent facial signals in conversation are eyebrow movements. Previous studies found links between eyebrow movements and questions, suggesting that these facial signals could contribute to the rapid recognition of questions. Therefore, we aimed to investigate whether early eyebrow movements (eyebrow frown or raise vs. no eyebrow movement) facilitate question identification. Participants were instructed to view videos of avatars where the presence of eyebrow movements accompanying questions was manipulated. Their task was to indicate whether the utterance was a question or a statement as accurately and quickly as possible. Data was collected using the online testing platform Gorilla. Results showed higher accuracies and faster response times for questions with eyebrow frowns, suggesting a facilitative role of eyebrow frowns for question identification. This means that facial signals can critically contribute to the communication of social actions in conversation by signaling social action-specific visual information, and provide visual cues to speakers’ intentions.

Interactionally Embedded Gestalt Principles of Multimodal Human Communication

“…Likewise, facial signals are integrated with speech not only to inform emotion recognition ( Pourtois et al, 2005 ) but also to complement speech and contribute to the semantic and pragmatic meaning of an utterance ( Bavelas & Chovil, 2018 ; Bavelas et al, 2014 ; Chovil, 1991 ; Domaneschi et al, 2017 ; Frith, 2009 ; Nota et al, 2021 , 2022 ). During natural conversation we produce hand gestures and facial signals alongside our speech that change how our utterance should be interpreted.…”

“…However, when we are in an interaction, the task is not just to understand the semantic content of an utterance but to very rapidly know how to respond. Part of what guides this will be in the percept itself (i.e., the ambient array of acoustic and visual energy; e.g., Benitez-Quiroz et al, 2016 ; Cienki, 2005 ; Domaneschi et al, 2017 ; Hömke et al, 2022 ; Nota et al, 2022 ; Zhang et al, 2021 ), but an important part of it will be in multiple levels of the interactional context: potential adjacency pairs, prior discourse, common ground, cultural norms, and so on. For example, as soon as we are aware that an utterance is a question, the type of response that is afforded is narrowed down, even if we do not know what the question is about or what our specific answer should be.…”

Virtual Reality: A tool to study language evolution

“…Theoretically, VR has proven particularly valuable for studying multimodal and multichannel communication (e.g., Chu & Hagoort, 2014;Hömke et al, 2018;Nota et al, 2022;Raghavan et al, 2023). Traditional studies often presented participants with only visual or auditory information, focusing on a single bodily channel, such as the voice or face.…”

Gestures speed up responses to questions

“…Moreover, visual signals produced earlier in the question may exert greater influence on fast responding (for such effects of eyebrow movements, see Nota et al, 2022 ). As the gesture timing spontaneously varied across questions (i.e.…”