Emotional State Recognition with Micro-expressions and Pulse Rate Variability

Abstract

Machine learning has known a tremendous growth within the last years, and lately, thanks to that, some computer vision algorithms started to access what is difficult or even impossible to perceive by the human eye. It is then natural that scientists began looking for ways to probe humans' emotions and their psyche with this technology. In this paper, we study the feasibility of recognizing and classifying the abstract concept of emotional states from videos of people facing a regular RGB camera. We do so by using the barely perceptible micro facial expressions humans cannot control, as well as the spontaneous variations of the pulse rate that we estimated using remote photoplethysmography. We compare these two modalities and our experimental results show that it is possible to classify emotional states from these implicit information gathered from regular cameras with encouraging performances.

Video-Based Pulse Rate Variability Measurement Using Periodic Variance Maximization and Adaptive Two-Window Peak Detection

“…The results showed that the accuracy of the classification is around 66%-83% for different stress levels. Belaiche et al [17] used both micro-expressions and PRV to predict three emotion states, namely, happiness, disgust and anger. This study found that although the accuracy of the PRV based method is higher than the micro-expressions based method in emotion states recognition but the average accuracy is usually not higher than 60% for the dataset that contains sudden emotion change.…”