Spatiotemporal Feature Descriptor for Micro-Expression Recognition Using Local Cube Binary Pattern

Abstract

Micro-expression recognition has been an active research area in recent years, it plays an important role in psychology and public security. Due to the aspects of short duration and subtle movement, it is challenging to extract spatiotemporal features of micro-expressions. The existing methods only extract features in the three-dimensional orthogonal plane and fail to make full use of that information. To solve this problem, we propose a new Local Cubes Binary Patterns (LCBP) method for micro-expression recognition. LCBP is cascaded by the motion information LCBP direction , the amplitude information LCBP amplitudes , and the spatial information LCBP 3D to obtain the spatiotemporal features. The advantage of LCBP is its ability to preserve the spatiotemporal information and the low feature dimension. Furthermore, to increase the discrimination of features in micro-expression sequences, we apply a differential calculation energy map to find regions of interest (ROI) for getting a weighted energy map. The final micro-expression feature acquired by fusing the LCBP features and the weighted energy map are classified through the Support Vector Machine (SVM). We evaluate the proposed method on four published micro-expression databases including SMIC, CASME, CASME2, SAMM. Experimental results demonstrate that our proposed method achieves promising performance for micro-expression recognition. INDEX TERMS Micro-expression recognition, differential energy map, local cubes binary patterns, SVM.

Sparse Spatiotemporal Descriptor for Micro-Expression Recognition Using Enhanced Local Cube Binary Pattern

“…Moreover, all of this spatiotemporal information, which manifests as changes in pixel brightness, is triggered by facial action. For the direction of facial action, our previous research of ME recognition demonstrates that facial actions are not limited to the temporal orthogonal planes [21]. Herein, we further use the ideas employed in our previous research to statistically distribute the temporal directions of the local facial action for different expression categories.…”

“…Guo et al [19] proposed to extend the Centralized Binary Pattern (CBP) features in three orthogonal planes to CBP-TOP descriptors, thereby obtaining lower-dimensional and more enriched representations. Meanwhile, due to the problem of losing information by encoding only in the three orthogonal planes, we extended the LBP encoding to the cube space and proposed Local Cube Binary Patterns (LCBP) for ME recognition [21]. LCBP concatenated the feature histograms of direction, amplitude, and central difference to obtain spatiotemporal features, which reduces the feature dimension while preserving the spatiotemporal information.…”

“…According to reports [21], the spatiotemporal variation in facial movements provides strong evidence for analyzing the emotional state of MEs, which are conveyed by the changes in pixel brightness of local image sequences. Clearly, LBP-TOP captures the key to predicting MEs by converting three-dimensional temporal changes into two-dimensional temporal planes, thereby effectively capturing time-domain characteristics using spatially encoded patterns.…”

“…Currently, advances in computer vision-based image classification, such as machine learning and deep learning algorithms, provide unprecedented opportunities for automated ME recognition. In this context, many handcrafted techniques and deep learning methods have been proposed to predict MEs using different feature extraction methods, yielding valuable insights into the progression patterns of ME recognition [10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28]. The mainstream feature representation methods for ME recognition are mainly based on the optical flow [13][14][15][16][17], local binary patterns (LBP) [18][19][20][21][22][23], and deep learning techniques [24][25][26][27][28].…”

“…Although the spatiotemporal features show improvements in ME recognition performance compared with the traditional LBP features, the indirect acquisition of LBP-TOP features may introduce noise, which cannot be ignored. Moreover, different variant studies based on LBP-TOP only reveal information related to the three orthogonal planes, which leads to the loss of some hidden clues [21]. Due to these facts, the representational capabilities in the time-domain is weakened to some extent by adopting the coding patterns of three orthogonal planes.…”