Emotion classification using a CNN_LSTM-based model for smooth emotional synchronization of the humanoid robot REN-XIN

Abstract

In this paper, we propose an Emotional Trigger System to impart an automatic emotion expression ability within the humanoid robot REN-XIN, in which the Emotional Trigger is an emotion classification model trained from our proposed Word Mover’s Distance(WMD) based algorithm. Due to the long time delay of the WMD-based Emotional Trigger System, we propose an enhanced Emotional Trigger System to enable a smooth interaction with the robot in which the Emotional Trigger is replaced by a conventional convolution neural network and a long short term memory network (CNN_LSTM)-based deep neural network. In our experiments, the CNN_LSTM based model only need 10 milliseconds or less to finish the classification without a decrease in accuracy, while the WMD-based model needed approximately 6-8 seconds to give a result. In this paper, the experiments are conducted based on the same sub-data sets of the Chinese emotional corpus(Ren_CECps) used in former WMD experiments: one comprises 50% data for training and 50% for testing(1v1 experiment), and the other comprises 80% data for training and 20% for testing(4v1 experiment). The experiments are conducted using WMD, CNN_LSTM, CNN and LSTM. The results show that CNN_LSTM obtains the best F1 score (0.35) in the 1v1 experiment and almost the same accuracy of F1 scores (0.366 vs 0.367) achieved by WMD in the 4v1 experiment. Finally, we present demonstration videos with the same scenario to show the performance of robot control driven by CNN_LSTM-based Emotional Trigger System and WMD-based Emotional Trigger System. To improve the comparison, total manual-control performance is also recorded.

Smile Like You Mean It: Driving Animatronic Robotic Face with Learned Models

“…Therefore, the approach cannot perform online inference, while our robot can react to a novel facial expression within 0.18s in a fully online manner. Liu and Ren [17] classified the input emotion and search in a discrete predefined expression set which constrains possible outcomes.…”

“…Some traditional methods [10]- [16] define a set of pre-specified facial expressions. Others generalize this process to search for closest match from a database [17] or by following an fitness function [17], [18]. However, as human expressions are highly diverse, these approaches have limited value in practical robot-human interactions.…”

A Review on Human-Computer Interaction and Intelligent Robots

“…Machine learning-based methods 243 Long-range iris recognition 244 Fingerprint recognition Fingerprint recognition for young children 245 Fingerprint recognition at crime scenes 246 Review works 247,248 Others Age and gender recognition 249,250 Facial expression recognition À À À CNN based methods 252,253 Multi-modality feature fusion-based method 254 Expression recognition based on static images 255 Micro-Expression Recognition [256][257][258] Facial expressions generation À À À Interactive GAN-based method 260 3D facial expression generation 261 Humanoid robot expression generation 23 Three-dimensional speaking characters 262 Expression generation natural description 264,265 Posture or gestures recognition À À À Driving posture recognition 266 Weighted fusion method for gesture recognition 267 Posture recognition for hazard prevention 268 Emotional body gesture recognition 269 Gesture recognition in video 271 Hand gesture recognition 272 deep neural network methods are introduced into the¯eld to seek a better recognition performance. The¯rst step of face recognition is the detection of face with an aim to determine whether faces exist on a given image or not.…”

Efeito da indução de emoções sobre a atenção visual em propagandas de cerveja: um estudo de neurociência aplicada ao marketing

“…O estudo das emoções tem atraído cada vez mais investigadores de diferentes disciplinas, tais como, psicologia (Burkitt, 2018) neurociência (Schweizer et al, 2019), inteligência artificial (Liu et al, 2019), economia e marketing (Rodrigues et al, 2019) apenas para citar alguns exemplos.…”