Specific facial signals associate with categories of social actions conveyed through questions

Abstract

The early recognition of fundamental social actions, like questions, is crucial for understanding the speaker’s intended message and planning a timely response in conversation. Questions themselves may express more than one social action category (e.g., an information request “What time is it?”, an invitation “Will you come to my party?” or a criticism “Are you crazy?”). Although human language use occurs predominantly in a multimodal context, prior research on social actions has mainly focused on the verbal modality. This study breaks new ground by investigating how conversational facial signals may map onto the expression of different types of social actions conveyed through questions. The distribution, timing, and temporal organization of facial signals across social actions was analysed in a rich corpus of naturalistic, dyadic face-to-face Dutch conversations. These social actions were: Information Requests, Understanding Checks, Self-Directed questions, Stance or Sentiment questions, Other-Initiated Repairs, Active Participation questions, questions for Structuring, Initiating or Maintaining Conversation, and Plans and Actions questions. This is the first study to reveal differences in distribution and timing of facial signals across different types of social actions. The findings raise the possibility that facial signals may facilitate social action recognition during language processing in multimodal face-to-face interaction.

Conversational Eyebrow Frowns Facilitate Question Identification: An Online Study Using Virtual Avatars

“…Questions with hand gestures occurring before or with questions appear to be associated with faster responses, which lead to speculations about mechanisms resulting in multimodal facilitation (ter Bekke et al, 2020;Holler, Kendrick, & Levinson, 2018). Thus, a similar effect may be observable for facial signals like eyebrow movements, especially since these often occur prior to or around the onset of the utterance (Kaukomaa, Peräkylä, & Ruusuvuori, 2014;Nota et al, 2021Nota et al, , 2022. Their early timing could help the addressee understand the speaker's intended message more quickly by speeding up recognition of the social action.…”

“…Information request questions served as the critical utterances, while statements were used as fillers. We selected early eyebrow movements (frowns or raises), since they most frequently occurred in the corpus (Nota et al, 2021(Nota et al, , 2022. These eyebrow movements had different onset times occurring before or at the start of the verbal utterance, and different durations.…”

“…There were another 20 additional fillers (10 questions and 10 statements) accompanied by facial signals of the eyes other than eyebrow movements (five eye widenings, five squints), to vary the visual signals and distract from the experimental aim. Since most facial signals frequently occurred before or at the start of the verbal utterance in the corpus (Nota et al, 2021(Nota et al, , 2022, we only selected information request questions and statements with eyebrow frowns, eyebrow raises, eye widenings, or squints that occurred before or at the onset of the utterance (mean onset difference between visual signals and verbal utterances: 536 ms). Within this time window, the facial signal movement onsets were variable.…”

“…An interesting avenue is to look at questions in particular, due to their fundamental role in conversation, and to investigate how facial signals may contribute to question identification. Therefore, in this study, we aim to investigate the contribution of eyebrow movements in questions, since they are prevalent facial signals in conversation (Nota et al, 2021(Nota et al, , 2022.…”

Turn-taking in human face-to-face interaction is multimodal: gaze direction and manual gestures aid the coordination of turn transitions

“…The speed of transitions between turns has also been shown to be affected by visual bodily actions: questions with gaze directed to the addressee and questions with gestures get faster responses [1,52], and visual signals often precede the onset of verbal utterances or corresponding semantic elements within them (e.g. [53][54][55][56][57]), thus equipping them with 'predictive potential' [44,54]. Despite these observations, however, the standard model of turn-taking for conversation remains primarily a vocal-auditory one.…”