Synthesising Facial Macro- and Micro-Expressions Using Reference Guided Style Transfer

Abstract

Long video datasets of facial macro- and micro-expressions remains in strong demand with the current dominance of data-hungry deep learning methods. There are limited methods of generating long videos which contain micro-expressions. Moreover, there is a lack of performance metrics to quantify the generated data. To address the research gaps, we introduce a new approach to generate synthetic long videos and recommend assessment methods to inspect dataset quality. For synthetic long video generation, we use the state-of-the-art generative adversarial network style transfer method—StarGANv2. Using StarGANv2 pre-trained on the CelebA dataset, we transfer the style of a reference image from SAMM long videos (a facial micro- and macro-expression long video dataset) onto a source image of the FFHQ dataset to generate a synthetic dataset (SAMM-SYNTH). We evaluate SAMM-SYNTH by conducting an analysis based on the facial action units detected by OpenFace. For quantitative measurement, our findings show high correlation on two Action Units (AUs), i.e., AU12 and AU6, of the original and synthetic data with a Pearson’s correlation of 0.74 and 0.72, respectively. This is further supported by evaluation method proposed by OpenFace on those AUs, which also have high scores of 0.85 and 0.59. Additionally, optical flow is used to visually compare the original facial movements and the transferred facial movements. With this article, we publish our dataset to enable future research and to increase the data pool of micro-expressions research, especially in the spotting task.

Self-Supervised Spontaneous Latent-Based Facial Expression Sequence Generation

“…Reference frames are often used to guide the facial generation process [44], [45]. These methods implement style transfer on available facial expression datasets to transfer the facial movements to a neutral face, guided by the reference frames.…”

3D-CNN for Facial Micro- and Macro-expression Spotting on Long Video Sequences using Temporal Oriented Reference Frame

“…These were used to facilitate ME expressions recognition [21,30]. With recent interest in ME and MaE spotting, researchers created long video datasets, SAMM Long Videos (SAMM-LV) [28,29] and CAS(ME) 2 [20], to better represent spontaneous emotion for ME and MaE spotting. This paper focuses on automated spotting of MaE and ME on SAMM-LV and CAS(ME) 2 .…”