Intra- and Inter-Contrastive Learning for Micro-expression Action Unit Detection

Abstract

Encoding facial expressions via Action Units (AUs) has been found effective for resolving the ambiguity issue among different expressions. In the literature, AU detection has extensive researches in macro-expressions. However, there is limited research about AU analysis for microexpressions (MEs). ME AU detection becomes a challenging problem because of the subtle facial motion. To alleviate this problem, in this paper, we study the contrastive learning for modeling subtle AUs and propose a novel ME AU detection method by learning the intra-and intercontrastive information among MEs. Through the intracontrastive learning module, the difference between the onset and apex frames is enlarged and utilized to obtain the discriminative representation for low-intensity AU detection. In addition, considering the subtle difference between ME AUs, the inter-contrastive learning is designed to automatically explore and enlarge the difference between different AUs to enhance the ME AU detection robustness. Intensive experiments on two widely used ME databases have demonstrated the effectiveness and generalization ability of our proposed method.

Facial Micro-Expressions: An Overview

“…Moreover, the AUs in MEs have imbalanced distribution, e.g., there are 129 AU4, while only 13 AU4 are in CASME II. Existing ME-AU detection research proposed to utilize the MaEs [194] or specific ME characteristics, such as subtle local facial movements [66], [195], [196], which are discussed in more detail in the following, to overcome these issues.…”

“…Other ME-AU research focuses on modeling subtle AUs [66], [195] based on ME characteristics. An intracontrastive and intercontrastive learning method was proposed to enlarge and utilize the contrastive information between the onset and apex frames to obtain the discriminative representation for low-intensity ME-AU detection [195].…”

“…Other ME-AU research focuses on modeling subtle AUs [66], [195] based on ME characteristics. An intracontrastive and intercontrastive learning method was proposed to enlarge and utilize the contrastive information between the onset and apex frames to obtain the discriminative representation for low-intensity ME-AU detection [195]. To effectively learn local facial movement and leverage relationship information between different facial regions to enhance the robustness of ME-AU detection, a spatial and channel attention module was designed to capture subtle ME-AUs by exploring high-order statistics [66].…”

Preface: Celebrating the 100th anniversary of chemical sciences in Nanjing University

“…Intuitively, the frame sequences within a video naturally offer the temporal evolution and consistency characteristics for the video objects, which have been extensively utilized as selfsupervisory signals [20]- [24]. However, the recent pervasive video-based contrastive learning methods usually learn videolevel representation [20]- [22], [22] or frame correspondence [23], [25], while AU representation should be frame-wisely discriminative within a video clip [26], [27] and consistent across identities that show analogous AUs [28], [29].…”