Rethinking Optical Flow Methods for Micro-Expression Spotting

Zhao, Yuan; Tong, Xin; Zhu, Zichong; Sheng, Jianda; Dai, Lei; Xu, Lei; Xia, Xuehai; Yu, Junsheng; Li, Jiao

doi:10.1145/3503161.3551602

Cited by 13 publications

(2 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Considering the computational time for ME spotting, Soh et al [22] proposed a many-core parallel LBP-TOP [23] algorithm to leveraging compute unified device architecture. Zhao et al [24] leveraged improved face alignment methods, more robust optical flow techniques, and superior facial landmark detectors. They employed a Bayesian optimization hybrid approach for optimizing parameters typically set manually.…”

Section: Feature Descriptor-based Methodsmentioning

confidence: 99%

FESNet: Spotting Facial Expressions Using Local Spatial Discrepancy and Multi-Scale Temporal Aggregation

Zhang,

Lu,

Wang

et al. 2024

cai

View full text Add to dashboard Cite

Facial expressions (FEs) spotting aims to split long videos into intervals of neutral expression, macro-expression, or micro-expression. Recent works mainly focus on feature descriptor or optical flow methods, suffering from difficulty capturing subtle facial motion and efficient temporal aggregation. This paper proposes a novel end-to-end network, named FESNet (Facial Expression Spotting Network), to solve the above challenges. The main idea is to model the subtle facial motion as local spatial discrepancy and incorporate temporal correlation by multi-scale temporal convolution. The FESNet comprises a local spatial discrepancy module (LSDM) and a multi-scale temporal aggregation module (MTAM). The LSDM first extracts the static spatial features from each frame by residual convolution and learns the inner spatial correlation by multi-head attention. Moreover, the subtle facial motion of facial expression is modeled as the discrepancy between the first frame and the current frame of the input interval, making frame-wise spatial proposals. Using the local spatial discrepancy features and proposals as input, the MTAM incorporates the temporal correlation by multi-scale temporal convolution and performs cascade refinement to make the final prediction. Furthermore, this paper proposes a smooth loss to ensure the temporal consistency of the cascade refined proposals from MTAM. Comprehensive experiments show that FESNet achieves competitive performance compared to state-of-the-art methods.

show abstract

Section: Feature Descriptor-based Methodsmentioning

confidence: 99%

FESNet: Spotting Facial Expressions Using Local Spatial Discrepancy and Multi-Scale Temporal Aggregation

Zhang,

Lu,

Wang

et al. 2024

cai

View full text Add to dashboard Cite

show abstract

“…Next, the current frame F i and frame F (i+k) (the k-th frame from the current frame F i ) were used to compute the optical flow features, where k is half of the average length of an expression interval. Because the TV-L1 optical flow estimation method is the most robust among all optical flow estimation methods tested in [29], we used it to compute the horizontal component u and vertical component v that consist of the first and second channel of the model input features. In addition, we used them to compute the optical strain , which catches subtle facial deformations from optical flow components [30]:…”

Section: Feature Extractionmentioning

confidence: 99%

SL-Swin: A Transformer-Based Deep Learning Approach for Macro- and Micro-Expression Spotting on Small-Size Expression Datasets

Chen

Zhong

2023

Electronics

View full text Add to dashboard Cite

In recent years, the analysis of macro- and micro-expression has drawn the attention of researchers. These expressions provide visual cues to an individual’s emotions, which can be used in a broad range of potential applications such as lie detection and policing. In this paper, we address the challenge of spotting facial macro- and micro-expression from videos and present compelling results by using a deep learning approach to analyze the optical flow features. Unlike other deep learning approaches that are mainly based on Convolutional Neural Networks (CNNs), we propose a Transformer-based deep learning approach that predicts a score indicating the probability of a frame being within an expression interval. In contrast to other Transformer-based models that achieve high performance by being pre-trained on large datasets, our deep learning model, called SL-Swin, which incorporates Shifted Patch Tokenization and Locality Self-Attention into the backbone Swin Transformer network, effectively spots macro- and micro-expressions by being trained from scratch on small-size expression datasets. Our evaluation outcomes surpass the MEGC 2022 spotting baseline result, obtaining an overall F1-score of 0.1366. Additionally, our approach performs well on the MEGC 2021 spotting task, with an overall F1-score of 0.1824 and 0.1357 on the CAS(ME)2 and SAMM Long Videos, respectively. The code is publicly available on GitHub.

show abstract

Local and Global Features Interactive Fusion Network for Macro- and Micro-expression Spotting in Long Videos

Xie,

2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Rethinking Optical Flow Methods for Micro-Expression Spotting

Cited by 13 publications

References 15 publications

FESNet: Spotting Facial Expressions Using Local Spatial Discrepancy and Multi-Scale Temporal Aggregation

FESNet: Spotting Facial Expressions Using Local Spatial Discrepancy and Multi-Scale Temporal Aggregation

SL-Swin: A Transformer-Based Deep Learning Approach for Macro- and Micro-Expression Spotting on Small-Size Expression Datasets

Local and Global Features Interactive Fusion Network for Macro- and Micro-expression Spotting in Long Videos

Contact Info

Product

Resources

About