Facial expression intensity estimation using label-distribution-learning-enhanced ordinal regression

Abstract

Facial expression intensity estimation has promising applications in health care and affective computing, such as monitoring patients' pain feelings. However, labeling facial expression intensity is a specialized and time-consuming task. To overcome the lack of labeled data, a variety of ordinal regression (OR) models have been presented to estimate the relative intensity of each expression image within a sequence in an unsupervised setting. However, these models cannot estimate absolute intensity without actual intensity labels. To overcome this problem, this paper introduces the label-distribution-learning-enhanced OR (LDL-EOR) approach for facial expression intensity estimation. LDL-EOR learns relative intensity with OR and absolute intensity by using the label distribution to resist the noisy labels caused by the biases of manual and automatic labeling. This design aims to improve the accuracy of absolute intensity estimation while keeping the cost of manual labeling low. The label distribution is converted into a continuous intensity value by calculating the mathematical expectation, which makes the prediction results meet both relative and absolute intensity constraints. Ensuring the feasibility of LDL-EOR in different supervised settings, this paper presents a unified label distribution generation framework to automatically relabel training data frame by frame. The generated soft labels are used to supervise the LDL-EOR model and enhance its robustness to the noise existing in the original labels. Numerous experiments were conducted on three public expression datasets (CK+, BU-4DFE, and PAIN) to validate the superiority of LDL-EOR relative to other state-of-the-art approaches.