Facial Expressions Recognition for Human–Robot Interaction Using Deep Convolutional Neural Networks with Rectified Adam Optimizer

Abstract

The interaction between humans and an NAO robot using deep convolutional neural networks (CNN) is presented in this paper based on an innovative end-to-end pipeline method that applies two optimized CNNs, one for face recognition (FR) and another one for the facial expression recognition (FER) in order to obtain real-time inference speed for the entire process. Two different models for FR are considered, one known to be very accurate, but has low inference speed (faster region-based convolutional neural network), and one that is not as accurate but has high inference speed (single shot detector convolutional neural network). For emotion recognition transfer learning and fine-tuning of three CNN models (VGG, Inception V3 and ResNet) has been used. The overall results show that single shot detector convolutional neural network (SSD CNN) and faster region-based convolutional neural network (Faster R-CNN) models for face detection share almost the same accuracy: 97.8% for Faster R-CNN on PASCAL visual object classes (PASCAL VOCs) evaluation metrics and 97.42% for SSD Inception. In terms of FER, ResNet obtained the highest training accuracy (90.14%), while the visual geometry group (VGG) network had 87% accuracy and Inception V3 reached 81%. The results show improvements over 10% when using two serialized CNN, instead of using only the FER CNN, while the recent optimization model, called rectified adaptive moment optimization (RAdam), lead to a better generalization and accuracy improvement of 3%-4% on each emotion recognition CNN.

Improving Human–Robot Interaction by Enhancing NAO Robot Awareness of Human Facial Expression

“…Although different, highly accurate CNN-based FER models have been developed in the literature over the last decade [ 22 ], their integration into a robotic platform was not yet proven effective. Of note, the CNN architecture reported by Melinte et al, which was successfully integrated into the NAO robotic platform and demonstrated to perform better than the proposed CNN-based FER model in the same task, was not actually evaluated in human–robot interaction conditions [ 43 ]. On the other hand, testing the CNN architecture developed and integrated into the robotic platform with human participants would allow one to assess its performance in real-world scenarios, paving the way for its implementation in robotics applications.…”

“…Firstly, a higher recognition rate for the angry and scared facial expressions should be achieved. Secondly the CNN’s architecture could be enhanced by employing, for instance, ResNet or Visual Geometry Group (VGG) model-based architectures, which have recently been shown to have great potential in this field [ 43 ]. Future research will focus on optimizing the CNN model in order to increase its performance, as well as testing it for complex human–robot interaction tasks such as those in the medical or neuropsychological fields.…”

The Analysis of Emotion Authenticity Based on Facial Micromovements

“…For a social robot to provide effective services, identifying the user’s intent is paramount. A recent human-robot interaction study applied deep neural networks to recognize a user’s facial expressions in real time [ 36 ]. Further recognition of the user’s false (e.g., deception) or pretended (e.g., to be polite) expressions might introduce more social, rich, and effective interactions.…”

Advanced Intelligent Control through Versatile Intelligent Portable Platforms

“…The effectiveness of several deep learning models of facial expression recognition, such as SSD and fast R-CNN for human–robot interaction is very interesting, deeply analysed and verified [ 10 ]. Based on an innovative end-to-end pipeline method that applies two optimized CNNs, the face recognition (FR) and the facial expression recognition (FER), using deep convolutional neural networks (CNN) for the growth real-time inference speed of the entire process is achieved, leading to a high level of advanced intelligent control in the interaction between humans and a NAO robot.…”