The influence of dynamics and speech on understanding humanoid facial expressions

Abstract

Human communication relies mostly on nonverbal signals expressed through body language. Facial expressions, in particular, convey emotional information that allows people involved in social interactions to mutually judge the emotional states and to adjust its behavior appropriately. First studies aimed at investigating the recognition of facial expressions were based on static stimuli. However, facial expressions are rarely static, especially in everyday social interactions. Therefore, it has been hypothesized that the dynamics inherent in a facial expression could be fundamental in understanding its meaning. In addition, it has been demonstrated that nonlinguistic and linguistic information can contribute to reinforce the meaning of a facial expression making it easier to be recognized. Nevertheless, few studies have been performed on realistic humanoid robots. This experimental work aimed at demonstrating the human-like expressive capability of a humanoid robot by examining whether the effect of motion and vocal content influenced the perception of its facial expressions. The first part of the experiment aimed at studying the recognition capability of two kinds of stimuli related to the six basic expressions (i.e. anger, disgust, fear, happiness, sadness, and surprise): static stimuli, that is, photographs, and dynamic stimuli, that is, video recordings. The second and third parts were focused on comparing the same six basic expressions performed by a virtual avatar and by a physical robot under three different conditions: (1) muted facial expressions, (2) facial expressions with nonlinguistic vocalizations, and (3) facial expressions with an emotionally neutral verbal sentence. The results show that static stimuli performed by a human being and by the robot were more ambiguous than the corresponding dynamic stimuli on which motion and vocalization were associated. This hypothesis has been also investigated with a 3-dimensional replica of the physical robot demonstrating that even in case of a virtual avatar, dynamic and vocalization improve the emotional conveying capability.

Voice in Human–Agent Interaction

“…Ghazali et al [60] found that facial expressions resulted in higher social agency scores compared to voice alone. Lazzeri et al [98] found that, regardless of virtual avatar or robot, "muted" facial expressions without vocalizations were confusing. Read and Belpaeme [150] found that nonverbal utterances conveyed classifiable emotion.…”

“…Findings show that people can perceive emotion through voice, if certain conditions are met. The inclusion of voice affect to visual expressions of affect appears key (e.g., References [74,98]), with some work suggesting that voice alone is more difficult to interpret (e.g., Reference [33]). For instance, if there is a mismatch between bodily affective cues and vocal affect, then people are confused (e.g., Reference [183]).…”

Sustainable Adaptation of New Technology—The Case of Humanoids Used for the Care of Older Adults

“…Research on robots concerns both the anthropomorphic features of robots and the expression of realistic behavior by robots. Lazzeri et al concentrated on the research of facial expressions performed by robot Eva because facial expressions convey emotional information that allows people to get involved in social interactions [44].…”

Designing Robots for Elderly from the Perspective of Potential End-Users: A Sociological Approach

“…Moreover, in the previous studies, the authors explained how age-friendly robots were successfully used for various applications, such as companioning, assisting, playing, and overcoming users’ inabilities [ 26 ]. The research has also indicated that interactions between robots and the elderly could be improved for better outcomes [ 41 ]. Especially people’s acceptance of robots, motivation to use electronic devices and enjoyment in daily routines [ 3 , 20 ].…”