Three-Dimensional, Kinematic, Human Behavioral Pattern-Based Features for Multimodal Emotion Recognition

Abstract

This paper presents a multimodal emotion recognition method that uses a feature-level combination of three-dimensional (3D) geometric features (coordinates, distance and angle of joints), kinematic features such as velocity and displacement of joints, and features extracted from daily behavioral patterns such as frequency of head nod, hand wave, and body gestures that represent specific emotions. Head, face, hand, body, and speech data were captured from 15 participants using an infrared sensor (Microsoft Kinect). The 3D geometric and kinematic features were developed using raw feature data from the visual channel. Human emotional behavior-based features were developed using inter-annotator agreement and commonly observed expressions, movements and postures associated to specific emotions. The features from each modality and the behavioral pattern-based features (head shake, arm retraction, body forward movement depicting anger) were combined to train the multimodal classifier for the emotion recognition system. The classifier was trained using 10-fold cross validation and support vector machine (SVM) to predict six basic emotions. The results showed improvement in emotion recognition accuracy (The precision increased by 3.28% and the recall rate by 3.17%) when the 3D geometric, kinematic, and human behavioral pattern-based features were combined for multimodal emotion recognition using supervised classification.

Limb contraction drives fear perception

“…This same analysis was 21 performed in a behavioural study with a larger stimulus set that included the current 16 video within-category similarity for the neutral condition. High within-category similarity was also 5 found for neutral in the majority of postural features. In the case of symmetry, this finding was 6 also accompanied by a high level of dissimilarity between neutral and the rest of the affective 7 movement categories and a within-and between-category similarity for anger and happiness.…”

“…Presumably, our brains transform this visual information into an . A recent body 5 of work has also involved this network in the processing of emotional body expressions. 6 Specifically, higher activity has been reported in several nodes of the AON for emotional body 7 actions as opposed to neutral ones (Grèzes, Pichon, & de Gelder, 2007; Kret, Pichon, Grèzes, Stimulation (TMS) studies have directly implicated both the IPL (Engelen, de Graaf, Sack, & 10 de Gelder, 2015) and the pSTS (Candidi, Stienen, Aglioti, & de Gelder, 2011) in the recognition 11 of fearful body expressions.…”

“…Experimental design, task and procedure 5 The (f)MRI data used in this paper was collected as part of another study. The original 6 experiment consisted of two experimental sessions, one presenting face and voice stimuli and 7 the other one body and voice stimuli.…”

“…4 Participants were given two seconds to press a button indicating the perceived colour. This was 5 performed to ensure that participants were paying attention to the task while not explicitly 6 directing their focus of attention to the explicit evaluation of the emotional expression. The total 7 duration of each functional run was eight minutes.…”

“…The second aim of the study was to determine whether (dis)similarities in body posture and/or 2 body-movement kinematics between different emotional categories could explain the neural 3 response at the whole-brain level and in body-selective regions of interest. Several areas were 4 defined known for their involvement in the processing of body expressions and their neural 5 RDMs were computed (see Methods and Supplementary Materials for more information 6 regarding ROI definition and location. For an inspection of the ROI matrices see Figure SR1).…”