Extended Local Binary Patterns for Efficient and Robust Spontaneous Facial Micro-Expression Recognition

Abstract

Facial Micro-Expressions (MEs) are spontaneous, involuntary facial movements when a person experiences an emotion but deliberately or unconsciously attempts to conceal his or her genuine emotions. Recently, ME recognition has attracted increasing attention due to its potential applications such as clinical diagnosis, business negotiation, interrogations, and security. However, it is expensive to build large scale ME datasets, mainly due to the difficulty of inducing spontaneous MEs. This limits the application of deep learning techniques which require lots of training data.In this paper, we propose a simple, efficient yet robust descriptor called Extended Local Binary Patterns on Three Orthogonal Planes (ELBPTOP) for ME recognition. ELBPTOP consists of three complementary binary descriptors: LBPTOP and two novel ones Radial Difference LBPTOP (RDLBPTOP) and Angular Difference LBPTOP (ADLBPTOP), which explore the local second order information along the radial and angular directions contained in ME video sequences. ELBPTOP is a novel ME descriptor inspired by unique and subtle facial movements. It is computationally efficient and only marginally increases the cost of computing LBPTOP, yet is extremely effective for ME recognition. In addition, by firstly introducing Whitened Principal Component Analysis (WPCA) to ME recognition, we can further obtain more compact and discriminative feature representations, then achieve significantly computational savings.Extensive experimental evaluation on three popular spontaneous ME datasets SMIC, CASME II and SAMM show that our proposed ELBPTOP approach significantly outperforms the previous state-of-the-art on all three single evaluated datasets and achieves promising results on cross-database recognition. Our code will be made available.Index Terms-Micro-expression recognition, local binary pattern, feature extraction. ADLBPTOP 2,4 + LBPTOP 3,4 + RDLBPXOT 3,4,1 Acc(%):73.94 ADLBPTOP 1,8 + LBPTOP 3,4 + RDLBPXOT 4,4,3 Acc(%):69.06 SMIC CASME II LBPTOP 3,4 Acc(%):71.55 ADLBPTOP 2,4 Acc(%):70.20 RDLBPXOT 3,4,1 Acc(%):69.24 LBPTOP 3,4 Acc(%):65.16 ADLBPTOP 1,8 Acc(%):63.73 RDLBPXOT 4,4,3 Acc(%):62.57

Sparse Spatiotemporal Descriptor for Micro-Expression Recognition Using Enhanced Local Cube Binary Pattern

“…Compared with the optical-flow-based methods Sparse MDMO [15] and Bi-WOOF+Phase [42], our method exhibits an accuracy improvement of 11.5% and 15.9% in CASME-II, respectively. The proposed method improved the accuracy of CASME-II by 4.51% and 8.35%, respectively, compared with the results reported by the recent handcrafted methods ELBPTOP [43] and LCBP [21]. This advantage is facilitated by the fact that the Enhanced LCBP consisted of spatial and temporal descriptors focused on capturing spatiotemporal information.…”

“…This work will contribute to the further exploration of sparse binary descriptors, which will improve the prediction of MEs, and thus combining the advantages of handcrafted features with deep learning technology will be our future work. ELBPTOP * [42] Extended Local Binary Patterns on Three Orthogonal Planes Bi-WOOF+Phase * [43] Bi-Weighted Oriented Optical Flow with phase information ELRCN * [24] Enriched Long-term Recurrent Convolutional Network 3D flow-based CNN * [26] 3D flow-based convolutional neural networks TSCNN * [28] Transferring Long-term Convolutional Neural Network SSSN * [44] Single-Stream Shallow Network DSSN * [44] Dual-Stream Shallow Network…”

“…Moreover, the combination of Multi-Regional Joint Sparse Learning allows the use of relevant information between different regions, which helps select more discriminating features. [14] 54.88% 56.14% 45.93% N/A MDMO * [13] 58.97% 56.29% 51.69% N/A Sparse MDMO * [15] 70.51% 74.83% 66.95% N/A HIGO * [8] 68.29% N/A 67.21% N/A STLBP-IP * [22] 57.93% N/A 59.51% N/A DISTLBP-RIP * [23] 63.41% 64.33% 64.78% N/A LCBP * [21] 71.30% 76.36% 70.10% 76.10% ELBPTOP * [43] 69.06% N/A 73.94% 63.44% Bi-WOOF+Phase * [42] 68.29% N/A 62.55% N/A Our Method 79.26% 78.23% 78.45% 79.41%…”

A New Feature Extraction and Recognition Method for Microexpression Based on Local Non-negative Matrix Factorization

“…We also adopt this approach. However, it is worth noting that Davison et al ( 2018a ) and Guo et al ( 2019 ) classify microexpressions using facial action units, instead of predicted emotions to remove the potential bias of human reporting.…”

“…We also adopt this approach. However, it is worth noting that Davison et al (2018a) and Guo et al (2019) classify microexpressions using facial action units, instead of predicted emotions to remove the potential bias of human reporting. Next, we will validate the proposed scheme based on CK+ macroexpression dataset (Kanade et al, 2000;Lucey et al, 2010), CASME2 (Yan et al, 2014), and SAMM (Davison et al, 2018a,b) microexpression datasets.…”