Dense prediction for micro-expression spotting based on deep sequence model

Abstract

Micro-expression (ME) analysis has been becoming an attractive topic recently. Nevertheless, the studies of ME mostly focus on the recognition task while spotting task is rarely touched. While micro-expression recognition methods have obtained the promising results by applying deep learning techniques, the performance of the ME spotting task still needs to be largely improved. Most of the approaches still rely upon traditional techniques such as distance measurement between handcrafted features of frames which are not robust enough in detecting ME locations correctly. In this paper, we propose a novel method for ME spotting based on a deep sequence model. Our framework consists of two main steps: 1) From each position of video, we extract a spatial-temporal feature that can discriminate MEs among extrinsic movements. 2) We propose to use a LSTM network that can utilize both local and global correlation of the extracted feature to predict the score of the ME apex frame. The experiments on two publicly databases of ME spotting demonstrate the effectiveness of our proposed method.

An Overview of Facial Micro-Expression Analysis: Data, Methodology and Challenge

“…Currently, LBP-TOP is reported by most of the existing ME datasets as the baseline evaluation. Due to its computational simplicity, LBP-TOP features are utilized in a variety of different MER frameworks including crossdomain MER and ME spotting [69], [70], [56].…”

“…Zhang et al [70] utilized both HOG-TOP and HIGO-TOP as the feature descriptors in their work of MER. Tran et al [56] built a spotting network based on LSTM using HOG-TOP and HIGO-TOP features as the input of the network. To better capture the structural changes in ME videos using Optical Flow Based Feature.…”

DynGeoNet: Fusion Network for Micro-expression Spotting

“…In [26], Zhang et al proposed using a Convolutional Neural Network (CNN) to detect the apex frame in two main steps: (1) constructing CNN networks to predict apex frames and neutral frames; (2) introducing a feature engineering technique to merge nearby detected samples. Tran et al [18] introduced the dense prediction-based technique by fusing spatial-temporal features with Long short-term memory architecture to calculate the apex score of the ME samples in long videos. Recently, Pan et al [13] proposed the bilinear convolutional neural network (LBCNN) to extract the local and global features of the face area for classifying the ME samples and macro-expression samples.…”

“…The studies in [12,16,18] raise the issue that the combination of handcrafted feature and deep learning technique is a reasonable approach to handle the problems of limited data. In these works, the handcrafted features are utilized as the first step to extract the discriminative information, then the robust tools from deep learning are used to learn the hybrid model for classification.…”

Sliding Window Based Micro-expression Spotting: A Benchmark

“…Tran et al[24] proposed another deep learning-based technique by combining spatial-temporal features with the deep sequence model to compute the apex score in long videos. Their framework consists of two main steps: 1)From each position of a video, they extract a spatial-temporal feature that can discriminate MEs among extrinsic movements.…”