Less is more: Micro-expression recognition from video using apex frame

Abstract

Despite recent interest and advances in facial micro-expression research, there is still plenty of room for improvement in terms of micro-expression recognition. Conventional feature extraction approaches for micro-expression video consider either the whole video sequence or a part of it, for representation. However, with the high-speed video capture of microexpressions (100-200 fps), are all frames necessary to provide a sufficiently meaningful representation? Is the luxury of data a bane to accurate recognition? A novel proposition is presented in this paper, whereby we utilize only two images per video, namely, the apex frame and the onset frame. The apex frame of a video contains the highest intensity of expression changes among all frames, while the onset is the perfect choice of a reference frame with neutral expression. A new feature extractor, Bi-Weighted Oriented Optical Flow (Bi-WOOF) is proposed to encode essential expressiveness of the apex frame. We evaluated the proposed method on five micro-expression databases-CAS(ME) 2 , CASME II, SMIC-HS, SMIC-NIR and SMIC-VIS. Our experiments lend credence to our hypothesis, with our proposed technique achieving a state-of-the-art F1-score recognition performance of 0.61 and 0.62 in the high frame rate CASME II and SMIC-HS databases respectively.

A Survey of Automatic Facial Micro-Expression Analysis: Databases, Methods, and Challenges

“…In their later work (Polikovsky and Kameda, 2013), a tracking algorithm is applied to track the facial points that had been manually detected at the first frame throughout the whole sequence. To prevent the hassle of manually detecting the facial points, majority of the recent works (Davison et al, 2015, 2016a,b; Liong et al, 2015, 2016b,c; Wang et al, 2016a; Xia et al, 2016) opt to apply automatic facial landmark detection. Instead of running the detection for the whole sequence of facial images, the facial points are only detected at the first frame and fixed in the consecutive frames with the assumption that these points will only change minimally due to the subtleness of MEs.…”

“…Unfortunately, areas of the face that are not useful for ME analysis such as the cheek area may still be captured within the triangular regions. To address this problem, more recent methods partition the face into a few region-of-interests (ROIs) (Polikovsky et al, 2009; Polikovsky and Kameda, 2013; Liong et al, 2015, 2016b,c, 2018; Davison et al, 2016b; Li et al, 2018). The ROIs are regions that correspond to one or more FACS action units (AUs).…”

“…As such, these regions contain rigid facial motions when the muscles (AUs) are activated. Some studies (Liong et al, 2015, 2016b,c; Davison et al, 2016b) show that ROIs are more effective compared to the use of the entire face in constraining the salient locations for spotting.…”

“…Besides spotting facial micro-movements, a few other works focused on spotting a specific type of ME phase, particularly the apex frame (Liong et al, 2015, 2016b,c; Yan and Chen, 2017). The apex frame, which is the instant indicating the most expressive emotional state in an ME sequence, is believed to be able to effectively reveal the true expression for the particular video.…”

“…Several works that focused on the spotting of the apex frame (Yan et al, 2014b; Liong et al, 2015, 2016b,c) used Mean Absolute Error (MAE) to compute how close are the estimated apex frames to the ground-truth apex frames:…”