Deep Learning for Micro-Expression Recognition: A Survey

Abstract

Micro-expressions (MEs) are involuntary facial movements revealing people's hidden feelings in high-stake situations and have practical importance in various fields. Early methods for Micro-expression Recognition (MER) are mainly based on traditional features. Recently, with the success of Deep Learning (DL) in various tasks, neural networks have received increasing interest in MER. Different from macro-expressions, MEs are spontaneous, subtle, and rapid facial movements, leading to difficult data collection and annotation, thus publicly available datasets are usually small-scale. Currently, various DL approaches have been proposed to solve the ME issues and improve MER performance. In this survey, we provide a comprehensive review of deep MER and define a new taxonomy for the field encompassing all aspects of MER based on DL, including datasets, each step of the deep MER pipeline, and performance comparisons of the most influential methods. The basic approaches and advanced developments are summarized and discussed for each aspect. Additionally, we conclude the remaining challenges and potential directions for the design of robust MER systems. Finally, ethical considerations in MER are discussed. To the best of our knowledge, this is the first survey of deep MER methods, and this survey can serve as a reference point for future MER research.

From Emotion AI to Cognitive AI

“…Currently, there are three mainstreams in artificial emotional intelligence, namely emotion recognition, emotion synthesis, and emotion augmentation [19]. Emotion recognition refers to applying computer systems to recognize human emotions in affective computing, and is one of the most traditional approaches that has been employed in emotion AI research [20]. Emotion recognition has been widely used in different modalities such as facial expressions [21−23], body gestures [24,25], acoustic or written linguistic content [26,27], physiological signals [28,29], and the fusion of multiple modalities [30][31][32].…”

pMATE: A Privacy-Preserving Map Retrieval Task Assignment Scheme in Spatial Crowdsourcing

“…With the increasingly powerful functions of intelligent mobile terminals as well as the convenience and high speed of network access, spatial crowdsourcing (SC) [1,2] as a new business cooperation model, becomes more and more popular and is widely used in urban services and data collection [3][4][5], such as Uber, Didi, OpenStreetMap, etc. Tere are three characteristics in SC tasks, respectively, the location, the required capabilities, and the deadline.…”

Learning two groups of discriminative features for micro-expression recognition

“…Micro-Expressions (MEs) can imply the true human emotions and help reveal the real psychological activities. Thus, MER has valuable potential applications [1], [2], such as medical diagnosis, emotion interfaces, security, and lie detection.…”

“…In recent years, deep learning methods are widely applied to MER, and present promising results [2]. These methods employed deep models to extract ME features from † Jinsheng Wei, Guanming Lu and Jingjie Yan ME videos or images.…”

“…These methods employed deep models to extract ME features from † Jinsheng Wei, Guanming Lu and Jingjie Yan ME videos or images. First, some works [2], [3] adopt the models with single stream structure to aggregate spatial or spatial-temporal information; second, in a lot of works, the models with two-stream or multi-stream structures [2], [4], [5] are designed to aggregate more feature information from different views; finally, the models with cascade structure are also effective in MER, such as Convolutional Neural Networks (CNN)+Graph Convolution Networks (GCN) [6].…”