Micro-expression detection in long videos using optical flow and recurrent neural networks

Abstract

Facial micro-expressions are subtle and involuntary expressions that can reveal concealed emotions. Microexpressions are an invaluable source of information in application domains such as lie detection, mental health, sentiment analysis and more. One of the biggest challenges in this field of research is the small amount of available spontaneous micro-expression data. However, spontaneous data collection is burdened by time-consuming and expensive annotation. Hence, methods are needed which can reduce the amount of data that annotators have to review. This paper presents a novel micro-expression spotting method using a recurrent neural network (RNN) on optical flow features. We extract Histogram of Oriented Optical Flow (HOOF) features to encode the temporal changes in selected face regions. Finally, the RNN spots short intervals which are likely to contain occurrences of relevant facial micro-movements. The proposed method is evaluated on the SAMM database. Any chance of subject bias is eliminated by training the RNN using Leave-One-Subject-Out cross-validation. Comparing the spotted intervals with the labeled data shows that the method produced 1569 false positives while obtaining a recall of 0.4654. The initial results show that the proposed method would reduce the video length by a factor of 3.5, while still retaining almost half of the relevant micro-movements. Lastly, as the model gets more data, it becomes better at detecting intervals, which makes the proposed method suitable for supporting the annotation process.

An Overview of Facial Micro-Expression Analysis: Data, Methodology and Challenge

“…Verburg et al [54] utilized Histogram of Oriented Optical Flow (HOOF) to encode the subtle changes in the time do-main for selected face regions. Li et al [92], [93] revisited the HOOF feature descriptor and proposed an enhanced version to reduce the redundant dimensions in HOOF.…”

“…There are many works that first extract spatial features among all frames, then use recurrent convolutional layer or LSTM module to explore their temporal correlation. Due to the small amount of training samples, many learning based works [86], [113], [54] include handcrafted features (e.g., optical flow, HOOF) to give a higher signal-to-noise (SNR) ratio in comparison to using raw pixel data.…”

“…The optical flow map and its related extension algorithms, e.g., HOOF, are commonly used for extracting motion deformations of facial regions and have shown good performance [54], [86], [113], [54]. RCN-F [86] first extracted an optical flow map from the located onset and apex frames followed by a recurrent convolutional network (RCN).…”

Shallow Optical Flow Three-Stream CNN For Macro- And Micro-Expression Spotting From Long Videos

“…The micro-expression is determined if the frame's feature vector is above the threshold set for peak detection. Most works utilise established pre-processing techniques involving landmark detection [7,8], region masking [5,9], and emphasis on specific facial regions via ROI selection [10,11,12]. Motion-based approaches can characterize the subtle movements on the face.…”

Micro-Expression Recognition Based On Video Motion Magnification And Pre-Trained Neural Network

“…Menkovski[5] only implemented RNN on the histogram of oriented optical flow features and encoded temporal movements within selected facial regions, reducing required computations and time to learn the model. Lei et al[6] used a graph-temporal convolutional network (Graph-TCN) to manage the graph structures developed on the facial landmarks, extracting both node and edge features from dual channels.…”