Micro Expression Recognition via Dual-Stream Spatiotemporal Attention Network

Abstract

Microexpression can manifest the real mood of humans, which has been widely concerned in clinical diagnosis and depression analysis. To solve the problem of missing discriminative spatiotemporal features in a small data set caused by the short duration and subtle movement changes of microexpression, we present a dual-stream spatiotemporal attention network (DSTAN) that integrates dual-stream spatiotemporal network and attention mechanism to capture the deformation features and spatiotemporal features of microexpression in the case of small samples. The Spatiotemporal networks in DSTAN are based on two lightweight networks, namely, the spatiotemporal appearance network (STAN) learning the appearance features from the microexpression sequences and the spatiotemporal motion network (STMN) learning the motion features from optical flow sequences. To focus on the discriminative motion areas of microexpression, we construct a novel attention mechanism for the spatial model of STAN and STMN, including a multiscale kernel spatial attention mechanism and global dual-pool channel attention mechanism. To obtain the importance of each frame in the microexpression sequence, we design a temporal attention mechanism for the temporal model of STAN and STMN to form spatiotemporal appearance network-attention (STAN-A) and spatiotemporal motion network-attention (STMN-A), which can adaptively perform dynamic feature refinement. Finally, the feature concatenate-SVM method is used to integrate STAN-A and STMN-A to a novel network, DSTAN. The extensive experiments on three small spontaneous microexpression data sets of SMIC, CASME, and CASME II demonstrate the proposed DSTAN can effectively cope with the recognition of microexpressions.

Deep Learning for Micro-Expression Recognition: A Survey

“…Moreover, Yao et al [174] learned the weights of each feature channel adaptively through adding squeezeand-andexcitation blocks. Additionally, recent works [56], [82], [83], [175] encoded the spatio-temporal and channel attention simultaneously to further boost the representational power of MEs. Specifically, CBAMNet [82] presented a convolutional block attention module (CBAM) cascading the spatial attention module (see Fig.…”

“…For the input, in general, the combined inputs can provide promising results on all of the datasets [70], [94], [175]. This is because the different input modalities can contribute information from different views.…”

“…On the basis of various input modalities, we can explore useful information on limited ME samples to the greatest extent. Since the combined inputs is a good choice for robust MER, the multi-stream network is recommended to learn effective representations from various inputs [70], [94], [175]. In contrast to the combined inputs, the sole sequence performs worse [172], due to the limited information and redundancy.…”

Recognition of Micro-Expressions using the Spatiotemporal Capsule Network

“…Handcrafted-based approaches include LBP-based [6], [15]- [20], optical-flow-based [21]- [24], and more techniques (e.g., colour space, histogram.). Using deep learning to extract additional handcrafted features [25]- [27] or fusing handcrafted features with deep-learning features, respectively [28]. Deep-learning-based may be separated into three categories depending on the input data: input onset frame and apex frame [31], which extract the face texture and light-shadow.…”

“…Yang et al [26] and bai et al [32] retrieved spatial features using VGGNet-16 and VGGFace, respectively, and then extracted temporal features using LSTM. Wang et al [27] extracted features from frame sequences and optical flow sequences, respectively, using STAN-A and STMN-A, and then conducted fusion classification. Lei et al [31] extracted characteristics between facial action units using graph convolution (AUs).…”