Causal Intervention for Subject-Deconfounded Facial Action Unit Recognition

Abstract

Subject-invariant facial action unit (AU) recognition remains challenging for the reason that the data distribution varies among subjects. In this paper, we propose a causal inference framework for subject-invariant facial action unit recognition. To illustrate the causal effect existing in AU recognition task, we formulate the causalities among facial images, subjects, latent AU semantic relations, and estimated AU occurrence probabilities via a structural causal model. By constructing such a causal diagram, we clarify the causal-effect among variables and propose a plug-in causal intervention module, CIS, to deconfound the confounder Subject in the causal diagram. Extensive experiments conducted on two commonly used AU benchmark datasets, BP4D and DISFA, show the effectiveness of our CIS, and the model with CIS inserted, CISNet, has achieved state-of-the-art performance.

Inferring Causal Factors of Core Affect Dynamics on Social Participation through the Lens of the Observer

“…Indeed, in real world cases, dependencies among time series are usually nonlinear and ignoring such interactions could lead to inconsistent estimation. However, using causal knowledge to improve machine learning algorithms remains an open area [ 98 , 100 ], and causal analysis in affective computing is at best in its infancy apart from a handful of exceptions (e.g., [ 100 , 101 , 102 , 103 , 104 , 105 ]).…”

Causal Structure Learning of Bias for Fair Affect Recognition

“…A natural extension of the above would be to attempt to leverage causal inference for another computer vision task: facial expression recognition. Indeed, existing attempts at doing so have been highly successful [2,29]. However, existing works have only leveraged on sequential data input [29] or investigated the use of interventions, back-door adjustments, and confounders [2,3].…”

“…Indeed, existing attempts at doing so have been highly successful [2,29]. However, existing works have only leveraged on sequential data input [29] or investigated the use of interventions, back-door adjustments, and confounders [2,3]. None of the existing works have explored the usage of structural causal models to formalise bias.…”

“…Though no specific causal mechanism was utilised, the authors imposed a "causal relation" by virtue of learning the relationship between past facial images to current affect states. On the other hand, Chen et al [2] addressed the AU recognition subject variation problem by removing the confounding effect caused by the confounder 'Subject'. Chen et al [3] addressed the dataset bias problem by proposing a network to induce backdoor adjustment in order to deconfound the dataset-related context features such as background scenes from the target emotion feature.…”