Impression Detection and Management Using an Embodied Conversational Agent

Abstract

Embodied Conversational Agents (ECAs) are a promising medium for human-computer interaction, since they are capable of engaging users in real-time face-to-face interaction [1, 2]. Users' formed impressions of an ECA (e.g. favour or dislike) could be reflected behaviourally [3,4]. These impressions may affect the interaction and could even remain afterwards [5,7]. Thus, when we build an ECA to impress users, it is important to detect how users feel about the ECA. The impression the ECA leaves can then be adjusted by controlling its non-verbal behaviour [7]. Motivated by the role of ECAs in interpersonal interaction and the state-of-the-art on affect recognition, we investigated three research questions: 1) which modality (facial expressions, eye movements, and physiological signals) reveals most of the formed impressions; 2) whether an ECA could leave a better impression by maximizing the impression it produces; 3) whether there are differences in impression formation during human-human vs. human-agent interaction. Our results firstly showed the interest to use different modalities to detect impressions. An ANOVA test indicated that facial expressions performance outperforms the physiological modality performance (M=1.27, p=0.02). Secondly, our results presented the possibility of creating an adaptive ECA. Compared with the randomly selected ECA behaviour, participants' ratings tended to be higher in the conditions where the ECA adapted its behaviour based on the detected impressions. Thirdly, we found similar behaviour during human-human vs. human-agent interaction. People treated an ECA similarly to a human by spending more time observing the face area when forming an impression.

Examining the Use of Nonverbal Communication in Virtual Agents

“…The agent would detect the user's affective state as well, progressing with the interview when appropriate cues were detected. Another example is one by Wang et al (2020), which featured a model that estimated users' impressions of an agent by monitoring their facial expressions and leveraging this information to change how users felt about the agent. Other agents recognize gaze behaviors through complex models derived from psychological research (Andrist et al, 2017;Huang et al, 2011) or trained using data from user studies (Morency et al, 2006).…”

Adaptation Mechanisms in Human–Agent Interaction: Effects on User’s Impressions and Engagement

“…These low-level signals are processed using EyesWeb and other external tools, such as machine learning pretrained models (Dermouche and Pelachaud, 2019;Wang et al, 2019), to extract high-level features about the user, such as their level of engagement.…”

“…The self-report annotations being considered separately, the MLP model was trained twice, one for warmth and one for competence. More details about this model can be found in the study by (Wang et al, 2019).…”