Facial Action Unit-based Deep Learning Framework for Spotting Macro- and Micro-expressions in Long Video Sequences

Yang, Bo; Wu, Jianming; Zhou, Zhiguang; Komiya, Megumi; Kishimoto, Koki; Xu, Jianfeng; Nonaka, Kenichi; Horiuchi, Toshiharu; Komorita, Satoshi; Hattori, Gen; Naito, Seiji; Takishima, Yasuhiro

doi:10.1145/3474085.3479209

Cited by 26 publications

(7 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Facial expressions can be divided into individual muscle movement components known as Action Units (AUs) [5]. As shown by the experiment described in [32], a single macro-or micro-expression may have more than one AU with high intensity.…”

Section: Swin Transformermentioning

confidence: 99%

See 1 more Smart Citation

SL-Swin: A Transformer-Based Deep Learning Approach for Macro- and Micro-Expression Spotting on Small-Size Expression Datasets

Chen

Zhong

2023

Electronics

View full text Add to dashboard Cite

In recent years, the analysis of macro- and micro-expression has drawn the attention of researchers. These expressions provide visual cues to an individual’s emotions, which can be used in a broad range of potential applications such as lie detection and policing. In this paper, we address the challenge of spotting facial macro- and micro-expression from videos and present compelling results by using a deep learning approach to analyze the optical flow features. Unlike other deep learning approaches that are mainly based on Convolutional Neural Networks (CNNs), we propose a Transformer-based deep learning approach that predicts a score indicating the probability of a frame being within an expression interval. In contrast to other Transformer-based models that achieve high performance by being pre-trained on large datasets, our deep learning model, called SL-Swin, which incorporates Shifted Patch Tokenization and Locality Self-Attention into the backbone Swin Transformer network, effectively spots macro- and micro-expressions by being trained from scratch on small-size expression datasets. Our evaluation outcomes surpass the MEGC 2022 spotting baseline result, obtaining an overall F1-score of 0.1366. Additionally, our approach performs well on the MEGC 2021 spotting task, with an overall F1-score of 0.1824 and 0.1357 on the CAS(ME)2 and SAMM Long Videos, respectively. The code is publicly available on GitHub.

show abstract

Section: Swin Transformermentioning

confidence: 99%

“…The existing macro-and micro-expression spotting approaches can be roughly divided into traditional approaches and deep learning approaches [5]. Traditional expression spotting approaches use manually crafted features to determine whether or not a frame is an expression frame.…”

Section: Introductionmentioning

confidence: 99%

SL-Swin: A Transformer-Based Deep Learning Approach for Macro- and Micro-Expression Spotting on Small-Size Expression Datasets

Chen

Zhong

2023

Electronics

View full text Add to dashboard Cite

show abstract

“…The framebased spotting task including onset, apex and offset location of one ME has been explored by Patel based on a discriminative response map fitting (DRMF) model [14].Guo considered the motion angle information and designed magnitude and angle combined (MAC) optical flow features to improve spotting efficiency [15]. In recent years, deep learning models have been widely used to detect the interval or apex frame of ME by using extracted features of raw images in video sequences as input [16]. Some interval-based deep learning models use video clips as input and utilize long short-term memory (LSTM) [17][18][19] or a clip proposal network [20] to obtain potential ME intervals.…”

Section: Introductionmentioning

confidence: 99%

Lite General Network and MagFace CNN for Micro-Expression Spotting in Long Videos

Yang²,

Cao³

et al. 2023

Preprint

View full text Add to dashboard Cite

Facial expressions, especially spontaneous micro-expressions, as an intuitive reflection of human emotions, have come through much concern along with rapid advances in computer vision recently. Micro-expressions are small in amplitude and short in duration and often appear together with macro-expressions, making micro-expression spotting in long videos as a challenging task. In this article, we propose intersection over minimum labelling method combined with a Lite General Network and MagFace CNN (LGNMNet) model to predict the possibility of video frames belonging to an micro-expression interval, which balances easy and difficult samples to improve the learning effect of training process. Experimental results show that our method achieves state-of-the-art performance in spotting micro-expressions in long videos of both the CAS(ME) 2 and SAMM-LV datasets (with F1-scores of 0.2474 and 0.2555, respectively). Additionally, a new pair-merge way of combining nearby detected apex frames to construct micro-expression intervals in post-processing stage has been devised and analysed, providing a feasible solution for the task of macro- and micro-spotting in long videos.

show abstract

“…Variant CNNbased models are also employed. For example, a Concat-CNN model consisting of three streams of convolutional networks with different sizes of convolution kernels [30] was proposed to learn feature correlations among facial action units (AUs) of different frames. In addition, a local bilinear convolutional neural network (LBCNN) [31] was proposed to transform the micro-expression spotting task into a fine-grained image recognition problem.…”

Section: Introductionmentioning

confidence: 99%

A Spatio-Temporal Spotting Network with Sliding Windows for Micro-Expression Detection

Fu,

An,

Huang

et al. 2023

Electronics

View full text Add to dashboard Cite

Micro-expressions reveal underlying emotions and are widely applied in political psychology, lie detection, law enforcement and medical care. Micro-expression spotting aims to detect the temporal locations of facial expressions from video sequences and is a crucial task in micro-expression recognition. In this study, the problem of micro-expression spotting is formulated as micro-expression classification per frame. We propose an effective spotting model with sliding windows called the spatio-temporal spotting network. The method involves a sliding window detection mechanism, combines the spatial features from the local key frames and the global temporal features and performs micro-expression spotting. The experiments are conducted on the CAS(ME)2 database and the SAMM Long Videos database, and the results demonstrate that the proposed method outperforms the state-of-the-art method by 30.58% for the CAS(ME)2 and 23.98% for the SAMM Long Videos according to overall F-scores.

show abstract

Facial Action Unit-based Deep Learning Framework for Spotting Macro- and Micro-expressions in Long Video Sequences

Cited by 26 publications

References 9 publications

SL-Swin: A Transformer-Based Deep Learning Approach for Macro- and Micro-Expression Spotting on Small-Size Expression Datasets

SL-Swin: A Transformer-Based Deep Learning Approach for Macro- and Micro-Expression Spotting on Small-Size Expression Datasets

Lite General Network and MagFace CNN for Micro-Expression Spotting in Long Videos

A Spatio-Temporal Spotting Network with Sliding Windows for Micro-Expression Detection

Contact Info

Product

Resources

About