Your Body Reveals Your Impressions about Others: A Study on Multimodal Impression Detection

Abstract

Formed impressions are crucial for human-human interaction (e.g. a job interview) and an interaction with a virtual agent/robot, since they can impact people's perceptions and willingness to be involved in the interaction. There are studies on how facial features (e.g. skin color, face shape), acoustic signals and non-verbal behaviors (e.g. gestures, postures) create/leave certain impressions. However there is little research focusing on how our bodies disclose our already formed impression of someone. Forming an impression leads to emotions and behaviors which can be measured. In this paper, we investigate recognition of evoked impression of warmth and competence from the nonverbal behaviors expressed by the person forming the impression. We conducted an experiment in which participants were watching impression stimuli. We measured participant's facial expressions, eye movements and physiological reactions (electrocardiography and galvanic skin response). To recognize impressions, we tested 2 multivariate regression models with the aforementioned multimodal recordings. Our best results demonstrate the possibility to detect impressions along warmth and competence dimensions with a concordance correlation coefficient of 0.838 and 0.864. Facial expressions and eye movements are more reliable for impression detection compared with physiological signals. Finally, the higher the Berkeley emotion expressivity scores the participants have, the more accurately the impressions are detected.

Impression Detection and Management Using an Embodied Conversational Agent

“…To the best of our knowledge, there is only one study on assessing the formed impressions from the body signals (e.g. facial expressions and gestures) of the person forming the impression [5]. In [5], it reported that formed impressions could be detected using multimodal signals in both multi-task and single task frameworks.…”

“…facial expressions and gestures) of the person forming the impression [5]. In [5], it reported that formed impressions could be detected using multimodal signals in both multi-task and single task frameworks. However, which body signal reveals the formed impression more expressively was not discussed.…”

“…1) and is the main focus of this paper. The impression expressive behaviour could be conveyed through multiple modalities, including facial, gestural and physiological reactions, which may not always be congruent and have the same level of importance [5]. To the best of our knowledge, there is rarely studies with ECA which measures users' impressions and adapts its behaviour accordingly.…”

“…In this paper, we would like to investigate three research questions: 1) which modality (facial, eye and physiological expressions) reveals most of the formed impressions; 2) whether an ECA could leave a better impression on users by maximizing the impression (W or C) it produces; 3) whether there are differences in impression formation during human-human Fig. 1: Impression Formation and Detection Diagram [5] vs. human-agent interaction. We first applied several impression detection models on each modality of an impression evoking corpus with continuous W&C self-reports.…”

“…favour or dislike) could be reflected behaviourally [3,4]. These impressions may affect the interaction and could even remain afterwards [5,7]. Thus, when we build an ECA to impress users, it is important to detect how users feel about the ECA.…”