Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns

Huang, Xiaohua; Zhao, Guoying; Hong, Xiaopeng; Zheng, Wenming; Pietikäinen, Matti

doi:10.1016/j.neucom.2015.10.096

Cited by 246 publications

(130 citation statements)

References 45 publications

Supporting

Mentioning

128

Contrasting

Unclassified

Order By: Relevance

“…Handcrafted Approach. Many pioneer ME research works [12,13,14,15,16,17] [57]. LBP is insensitive to illumination change, computational simplicity, capable of handling a variety of spatial information, robust to rotation and translation.…”

Section: Feature Extractionmentioning

confidence: 99%

Evaluation of the Spatio-Temporal Features and GAN for Micro-Expression Recognition System

Liong

Gan

Zheng

et al. 2020

J Sign Process Syst

View full text Add to dashboard Cite

Owing to the development and advancement of artificial intelligence, numerous works were established in the human facial expression recognition system. Meanwhile, the detection and classification of micro-expressions are attracting attentions from various research communities in the recent few years. In this paper, we first review the processes of a conventional optical-flow-based recognition system, which comprised of facial landmarks annotations, optical flow guided images computation, features extraction and emotion class categorization. Secondly, a few approaches have been proposed to improve the feature extraction part, such as exploiting GAN to generate more image samples. Particularly, several variations of optical flow are computed in order to generate optimal images to lead to high recognition accuracies. Next, GAN, a combination of Generator and Discriminator, is utilized to generate new "fake" images to increase the sample size. Thirdly, a modified state-of-the-art convolutional neural networks is proposed. To verify the effectiveness of the the proposed method, the results are evaluated on spontaneous micro-expression databases, namely SMIC, CASME II and SAMM. Both the F1-score and accuracy performance metrics are reported in this paper.

show abstract

Section: Feature Extractionmentioning

confidence: 99%

Evaluation of the Spatio-Temporal Features and GAN for Micro-Expression Recognition System

Liong

Gan

Zheng

et al. 2020

J Sign Process Syst

View full text Add to dashboard Cite

show abstract

“…The performance of LBP-MOP was comparable to LBP-SIP, but with its computation time dramatically reduced. While LBP considers only pixel intensities, spatio-temporal completed local quantized patterns (STCLQP) (Huang et al, 2016) exploited more information containing sign, magnitude, and orientation components. To address the sparseness problem (in most LBP variants), specific codebooks were designed to reduce the number of possible codes to achieve better compactness.…”

Section: Recognition Of Facial Micro-expressionsmentioning

confidence: 99%

A Survey of Automatic Facial Micro-Expression Analysis: Databases, Methods, and Challenges

et al. 2018

View full text Add to dashboard Cite

Over the last few years, automatic facial micro-expression analysis has garnered increasing attention from experts across different disciplines because of its potential applications in various fields such as clinical diagnosis, forensic investigation and security systems. Advances in computer algorithms and video acquisition technology have rendered machine analysis of facial micro-expressions possible today, in contrast to decades ago when it was primarily the domain of psychiatrists where analysis was largely manual. Indeed, although the study of facial micro-expressions is a well-established field in psychology, it is still relatively new from the computational perspective with many interesting problems. In this survey, we present a comprehensive review of state-of-the-art databases and methods for micro-expressions spotting and recognition. Individual stages involved in the automation of these tasks are also described and reviewed at length. In addition, we also deliberate on the challenges and future directions in this growing field of automatic facial micro-expression analysis.

show abstract

“…[37] explores two effective binary face descriptors: Hot Wheel Patterns [37] and Dual-Cross Patterns [38] and makes use of abundant labelled micro-expressions. Besides computing the sign of pixel differences, Spatio-Temporal Completed Local Quantized Patterns (STCLQP) [39] also exploits the complementary components of magnitudes and orientations. Decorrelated Local Spatiotemporal Directional Features (DLSTD) [40] uses Robust Principal Component Analysis (RPCA) [41] to extract subtle emotion information and division of 16 Regions of Interest (ROIs) to utilize the Action Unit (AU) information.…”

Section: Related Workmentioning

confidence: 99%

Extended Local Binary Patterns for Efficient and Robust Spontaneous Facial Micro-Expression Recognition

et al. 2019

View full text Add to dashboard Cite

Facial Micro-Expressions (MEs) are spontaneous, involuntary facial movements when a person experiences an emotion but deliberately or unconsciously attempts to conceal his or her genuine emotions. Recently, ME recognition has attracted increasing attention due to its potential applications such as clinical diagnosis, business negotiation, interrogations, and security. However, it is expensive to build large scale ME datasets, mainly due to the difficulty of inducing spontaneous MEs. This limits the application of deep learning techniques which require lots of training data.In this paper, we propose a simple, efficient yet robust descriptor called Extended Local Binary Patterns on Three Orthogonal Planes (ELBPTOP) for ME recognition. ELBPTOP consists of three complementary binary descriptors: LBPTOP and two novel ones Radial Difference LBPTOP (RDLBPTOP) and Angular Difference LBPTOP (ADLBPTOP), which explore the local second order information along the radial and angular directions contained in ME video sequences. ELBPTOP is a novel ME descriptor inspired by unique and subtle facial movements. It is computationally efficient and only marginally increases the cost of computing LBPTOP, yet is extremely effective for ME recognition. In addition, by firstly introducing Whitened Principal Component Analysis (WPCA) to ME recognition, we can further obtain more compact and discriminative feature representations, then achieve significantly computational savings.Extensive experimental evaluation on three popular spontaneous ME datasets SMIC, CASME II and SAMM show that our proposed ELBPTOP approach significantly outperforms the previous state-of-the-art on all three single evaluated datasets and achieves promising results on cross-database recognition. Our code will be made available.Index Terms-Micro-expression recognition, local binary pattern, feature extraction. ADLBPTOP 2,4 + LBPTOP 3,4 + RDLBPXOT 3,4,1 Acc(%):73.94 ADLBPTOP 1,8 + LBPTOP 3,4 + RDLBPXOT 4,4,3 Acc(%):69.06 SMIC CASME II LBPTOP 3,4 Acc(%):71.55 ADLBPTOP 2,4 Acc(%):70.20 RDLBPXOT 3,4,1 Acc(%):69.24 LBPTOP 3,4 Acc(%):65.16 ADLBPTOP 1,8 Acc(%):63.73 RDLBPXOT 4,4,3 Acc(%):62.57

show abstract

Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns

Cited by 246 publications

References 45 publications

Evaluation of the Spatio-Temporal Features and GAN for Micro-Expression Recognition System

Evaluation of the Spatio-Temporal Features and GAN for Micro-Expression Recognition System

A Survey of Automatic Facial Micro-Expression Analysis: Databases, Methods, and Challenges

Extended Local Binary Patterns for Efficient and Robust Spontaneous Facial Micro-Expression Recognition

Contact Info

Product

Resources

About