Using a Social Robot to Evaluate Facial Expressions in the Wild

Abstract

In this work an affective computing approach is used to study the human-robot interaction using a social robot to validate facial expressions in the wild. Our global goal is to evaluate that a social robot can be used to interact in a convincing manner with human users to recognize their potential emotions through facial expressions, contextual cues and bio-signals. In particular, this work is focused on analyzing facial expression. A social robot is used to validate a pre-trained convolutional neural network (CNN) which recognizes facial expressions. Facial expression recognition plays an important role in recognizing and understanding human emotion by robots. Robots equipped with expression recognition capabilities can also be a useful tool to get feedback from the users. The designed experiment allows evaluating a trained neural network in facial expressions using a social robot in a real environment. In this paper a comparison between the CNN accuracy and human experts is performed, in addition to analyze the interaction, attention and difficulty to perform a particular expression by 29 non-expert users. In the experiment, the robot leads the users to perform different facial expressions in motivating and entertaining way. At the end of the experiment, the users are quizzed about their experience with the robot. Finally, a set of experts and the CNN classify the expressions. The obtained results allow affirming that the use of social robot is an adequate interaction paradigm for the evaluation on facial expression.

Artificial Vision Algorithms for Socially Assistive Robot Applications: A Review of the Literature

“…Regarding the computer resources used to recognize facial expressions, two implementations are reported using basic hardware computers [ 34 , 39 ] and computers with graphic cards [ 30 , 31 ]. It is worth noting the use of robotic platforms such as the NAO [ 27 , 28 , 33 , 38 ], R-50 Alice [ 35 , 40 ], Pepper [ 32 ], kiwi [ 44 ], and N-Maria [ 42 ].…”

“…Another example is presented by Ramis et al in [ 33 ]. The authors proposed an algorithm to recognize the emotions of people using the NAO robot, employing a game: (a) the VJ algorithm carries out face detection; (b) facial landmarks are located to calculate the center of each eye and the distance between them; (c) a CNN processes the image to determine whether the expression relates to happiness, sadness, disgust, anger, surprise, fear, or indifference.…”

Hybrid Method of Automated EEG Signals’ Selection Using Reversed Correlation Algorithm for Improved Classification of Emotions

“…In the past decades, many researchers have investigated subjects’ reactions and potentials of different emotion recognition techniques, including speech, non-verbal audition, facial expression, visual and thermal images, peripheral neural signals, and central neural system signals [ 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 ].…”

A Survey of AI-Based Facial Emotion Recognition: Features, ML & DL Techniques, Age-Wise Datasets and Future Directions

“…Today the need for emotion classification has surpassed the barrier of age because of the global shift towards online platforms such as online education to teach or gain knowledge virtually globally to all remote areas, IoT enabled health monitoring systems and temperature setters in cars and households, robotics, psychiatric evaluation based on violent behaviors of criminals or those mentally disturbed, mood swings study on adolescents to help guide them mentally, deepfake detection, gaming, and many such applications are currently being innovated using state of the art technologies. Also, numerous studies have been conducted on Facial Emotion Recognition by using Computer vision because of its practicality in intelligent robotics, health-related treatment, IoT, Security surveillance, criminal psychological analysis, observation of driver exhaustion, and other human-computer interfaces mechanisms [7][8][9]. With more virtual connectivity through videos and images, the need to adopt the latest technology based on people's emotions is now a critical factor in driving user-friendliness and maximum user satisfaction.…”