A Boost in Revealing Subtle Facial Expressions: A Consolidated Eulerian Framework

Abstract

Facial Micro-expression Recognition (MER) distinguishes the underlying emotional states of spontaneous subtle facial expressions. Automatic MER is challenging because that 1) the intensity of subtle facial muscle movement is extremely low and 2) the duration of ME is transient. Recent works adopt motion magnification or time interpolation to resolve these issues. Nevertheless, existing works divide them into two separate modules due to their non-linearity. Though such operation eases the difficulty in implementation, it ignores their underlying connections and thus results in inevitable losses in both accuracy and speed. Instead, in this paper, we explore their underlying joint formulations and propose a consolidated Eulerian framework to reveal the subtle facial movements. It expands the temporal duration and amplifies the muscle movements in micro-expressions simultaneously. Compared to existing approaches, the proposed method can not only process ME clips more efficiently but also make subtle ME movements more distinguishable. Experiments on two public MER databases indicate that our model outperforms the state-of-the-art in both speed and accuracy.

Facial Micro-Expression Recognition Using Two-Dimensional Landmark Feature Maps

“…For fair evaluation, we used a protocol called leave-onesubject-out cross validation (LOSO) in the same way as the conventional methods [26][27][28][29][30]. In LOSO, samples for one subject are used as test data, and samples for the remaining subjects are adopted as training data.…”

“…They extracted the atomic feature representing the region of interest from the optical flow information, and applied sparse coding, and then classified the result using SVM. Peng et al proposed a consolidated Eulerian frame that integrated independent motion magnification and frame interpolation into a single process [30]. Guo et al proposed the extended local binary patterns on three orthogonal planes (ELBPTOP) as feature descriptors for recognizing FME [45].…”

Learning from Macro-expression

“…Since standard CNNs are limited by their weakness in representing part-global relation, Nguyen et al [34] adopted the newly proposed framework CapsuleNet [30] to recognize micro-expressions. Peng et al [28] explored the underlying joint formulation for Motion MAGnification (MAG) and Time Interpolation Model (TIM) and proposed a consolidated framework for revealing the spatial-temporal information in micro-expression clips. Khor et al [11] presented a Dual-Stream Shallow Network (DSSN) which robustly learns deep micro-expression features by exploiting a pair of shallow CNNs with heterogeneous motion-based inputs.…”

“…We compare our framework with other related works. These methods are: 1) LBP-TOP [13], LBP-SIP [35], STLBP-IP [8], STCLQP [9], which are LBP based methods 2) HIGO [15], FHOFO [7], Bi-WOOF [20], which are optimal flow based methods, 3) Only-Apex [17], OFF-Apex [5], CNN+LSTM [12], Boost [28], DSSN [11], Shallow [19], Dual [40] Capsule [34], which are deep feature methods, 4) Dynamic [32], which is action unit assisted method. and 5) Neural [21], which is macro-expression assisted method.…”

Video Action Recognition Via Neural Architecture Searching

“…Video action recognition [1], which is a hot topic of video analysis and understanding, has drawn considerable attention from both academia and industry, since it has great value to many potential applications, like behaviour analysis [2], security, and video affective computing [3]. On one hand, new and large-scale datasets, such as Kinetics [4], Something-Something [5], make great contribution to video action recognition.…”