Recognizing Spontaneous Micro-Expression Using a Three-Stream Convolutional Neural Network

Abstract

Micro-expression recognition (MER) has attracted much attention with various practical applications, particularly in clinical diagnosis and interrogations. In this paper, we propose a three-stream convolutional neural network (TSCNN) to recognize MEs by learning ME-discriminative features in three key frames of ME videos. We design a dynamic-temporal stream, static-spatial stream, and local-spatial stream module for the TSCNN that respectively attempt to learn and integrate temporal, entire facial region, and facial local region cues in ME videos with the goal of recognizing MEs. In addition, to allow the TSCNN to recognize MEs without using the index values of apex frames, we design a reliable apex frame detection algorithm. Extensive experiments are conducted with five public ME databases: CASME II, SMIC-HS, SAMM, CAS(ME) 2 , and CASME. Our proposed TSCNN is shown to achieve more promising recognition results when compared with many other methods. INDEX TERMS Micro-expression recognition, convolutional neural networks, apex frame location, spatiotemporal information.

Sparse Spatiotemporal Descriptor for Micro-Expression Recognition Using Enhanced Local Cube Binary Pattern

“…This work will contribute to the further exploration of sparse binary descriptors, which will improve the prediction of MEs, and thus combining the advantages of handcrafted features with deep learning technology will be our future work. ELBPTOP * [42] Extended Local Binary Patterns on Three Orthogonal Planes Bi-WOOF+Phase * [43] Bi-Weighted Oriented Optical Flow with phase information ELRCN * [24] Enriched Long-term Recurrent Convolutional Network 3D flow-based CNN * [26] 3D flow-based convolutional neural networks TSCNN * [28] Transferring Long-term Convolutional Neural Network SSSN * [44] Single-Stream Shallow Network DSSN * [44] Dual-Stream Shallow Network…”

“…Yang et al [27] proposed a cascade structure of three VGG-NETs and a Long Short-Term Memory (LSTM) network LSTM, and integrated three different attention models in the spatial-domain. Song et al [28] developed a Three-stream Convolutional Neural Network (TSCNN) by encoding the discriminative representations in three key frames. TSCNN consisted of a dynamic-temporal stream, static-spatial stream, and local-spatial stream module, which are used to capture the characteristics of temporal, entire facial region, and local facial region, respectively.…”

“…Currently, advances in computer vision-based image classification, such as machine learning and deep learning algorithms, provide unprecedented opportunities for automated ME recognition. In this context, many handcrafted techniques and deep learning methods have been proposed to predict MEs using different feature extraction methods, yielding valuable insights into the progression patterns of ME recognition [10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28]. The mainstream feature representation methods for ME recognition are mainly based on the optical flow [13][14][15][16][17], local binary patterns (LBP) [18][19][20][21][22][23], and deep learning techniques [24][25][26][27][28].…”

“…Constrained by the sensitivity to the illumination, most of the optical-flow-based studies suffer from the interference of background noise, which limits their applicability. In recent years, the deep learning technology, with a rapid revival, has made it possible to avoid complicated handcrafted features design [28]. However, the expected performance is achieved by requiring a large number of training samples.…”

Fingerprint Presentation Attack Detection Utilizing Spatio-Temporal Features

“…The method had been proposed with application to facial expression recognition and reported over 95% accuracy. Moreover, a recent study on spontaneous facial micro-expression recognition suggested a deep learning model based on spatial and temporal streams and reported 63.53–74.05% accuracy [ 21 ].…”