Investigating Bias and Fairness in Facial Expression Recognition

Xu, Tian; White, Jennifer C.; Kalkan, Sinan; Güneş, Hatice

doi:10.48550/arxiv.2007.10075

“…More specific to faces, Drozdowski et al summarizes that the cohorts of concern in biometrics are demographic (e.g., sex, age, and race), person-specific (e.g., pose or expression [46], and accessories like eye-wear or makeup), and environmental (e.g., camera-model, sensor size, illumination, occlusion) [5]. Albiero et al found empirical support that having training data that is well balanced in gender does not mean that results of a genderbalanced test set will be balanced [47].…”

Section: Imbalanced Data and Data Problems In Frmentioning

confidence: 99%

Balancing Biases and Preserving Privacy on Balanced Faces in the Wild

Robinson¹,

Qin²,

Henon³

et al. 2021

Preprint

View full text Add to dashboard Cite

There are demographic biases in the SOTA CNN used for FR. Our BFW dataset serves as a proxy to measure bias across ethnicity and gender subgroups, allowing us to characterize FR performances per subgroup. We show performances are non-optimal when a single score threshold is used to determine whether sample pairs are genuine or imposter. Furthermore, actual performance ratings vary greatly from the reported across subgroups. Thus, claims of specific error rates only hold true for populations matching that of the validation data. We mitigate the imbalanced performances using a novel domain adaptation learning scheme on the facial encodings extracted using SOTA deep nets. Not only does this technique balance performance, but it also boosts the overall performance. A benefit of the proposed is to preserve identity information in facial features while removing demographic knowledge in the lower dimensional features. The removal of demographic knowledge prevents future potential biases from being injected into decision making. Additionally, privacy concerns are satisfied by this removal. We explore why this works qualitatively with hard samples. We also show quantitatively that subgroup classifiers can no longer learn from the encodings mapped by the proposed.

show abstract

“…In this survey, human bias, as opposed to machine learning algorithm, was the topic of investigation. More directly related to our research field is Xu et al (2020). In their research, the authors investigate bias in facial expression recognition, albeit without the consideration of Action Units.…”

Section: Biasmentioning

confidence: 99%

Uncovering the Bias in Facial Expressions

Deuschel,

Finzel,

Rieger

2021

Forschende Frauen in Bamberg

2

0

View full text Add to dashboard Cite

Over the past decades the machine and deep learning community has celebrated great achievements in challenging tasks such as image classification. The deep architecture of artificial neural networks together with the plenitude of available data makes it possible to describe highly complex relations. Yet, it is still impossible to fully capture what the deep learning model has learned and to verify that it operates fairly and without creating bias, especially in critical tasks, for instance those arising in the medical field. One example for such a task is the detection of distinct facial expressions, called Action Units, in facial images. Considering this specific task, our research aims to provide transparency regarding bias, specifically in relation to gender and skin color. We train a neural network for Action Unit classification and analyze its performance quantitatively based on its accuracy and qualitatively based on heatmaps. A structured review of our results indicates that we are able to detect bias. Even though we cannot conclude from our results that lower classification performance emerged solely from gender and skin color bias, these biases must be addressed, which is why we end by giving suggestions on how the detected bias can be avoided.

show abstract

Uncovering the Bias in Facial Expressions

Deuschel¹,

Finzel²,

Rieger³

2021

Preprint

2

0

View full text Add to dashboard Cite

Over the past decades the machine and deep learning community has celebrated great achievements in challenging tasks such as image classification. The deep architecture of artificial neural networks together with the plenitude of available data makes it possible to describe highly complex relations. Yet, it is still impossible to fully capture what the deep learning model has learned and to verify that it operates fairly and without creating bias, especially in critical tasks, for instance those arising in the medical field. One example for such a task is the detection of distinct facial expressions, called Action Units, in facial images. Considering this specific task, our research aims to provide transparency regarding bias, specifically in relation to gender and skin color. We train a neural network for Action Unit classification and analyze its performance quantitatively based on its accuracy and qualitatively based on heatmaps. A structured review of our results indicates that we are able to detect bias. Even though we cannot conclude from our results that lower classification performance emerged solely from gender and skin color bias, these biases must be addressed, which is why we end by giving suggestions on how the detected bias can be avoided.

show abstract

Investigating Bias and Fairness in Facial Expression Recognition

Cited by 4 publications

References 45 publications

Balancing Biases and Preserving Privacy on Balanced Faces in the Wild

Balancing Biases and Preserving Privacy on Balanced Faces in the Wild

Uncovering the Bias in Facial Expressions

Uncovering the Bias in Facial Expressions

Contact Info

Product

Resources

About