Comparing the Performance of Facial Emotion Recognition Systems on Real-Life Videos: Gender, Ethnicity and Age

Ahmad, Khurshid; Wang, Shirui; Vogel, Carl; Jain, Pranav; O’Neill, Oscar; Sufi, Basit Hamid

doi:10.1007/978-3-030-89906-6_14

“…Demographically speaking, some datasets openly focus on facial expressions of specific demographic groups, such as JAFFE [46], [47] (Japanese women) and iSAFE [48] (Indian people), but for the most part the datasets have not been collected taking diversity into account. Some recent works have already found specific biases around gender, race, and age in both commercial FER systems [49], [50] and research models [51], [52], [53], [54], [55], [56]. From these works, Kim et al [49] focus on the age bias of commercial models in an age-labeled dataset.…”

Section: Facial Expression Recognitionmentioning

confidence: 99%

“…From these works, Kim et al [49] focus on the age bias of commercial models in an age-labeled dataset. Ahmad et al [50] also study commercial models, but extend the research to age, gender, and race (considering two racial categories) by employing a custom database of politician videos. Regarding research models, Xu et al [52] study age, gender, and race bias in models trained on an Internet search-gathered dataset and evaluated on a different dataset with known demographic characteristics.…”

Section: Facial Expression Recognitionmentioning

confidence: 99%

Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition

Dominguez-Catena,

Paternain,

Galar

2024

IEEE Trans. Pattern Anal. Mach. Intell.

4

0

View full text Add to dashboard Cite

Demographic biases in source datasets have been shown as one of the causes of unfairness and discrimination in the predictions of Machine Learning models. One of the most prominent types of demographic bias are statistical imbalances in the representation of demographic groups in the datasets. In this paper, we study the measurement of these biases by reviewing the existing metrics, including those that can be borrowed from other disciplines. We develop a taxonomy for the classification of these metrics, providing a practical guide for the selection of appropriate metrics. To illustrate the utility of our framework, and to further understand the practical characteristics of the metrics, we conduct a case study of 20 datasets used in Facial Emotion Recognition (FER), analyzing the biases present in them. Our experimental results show that many metrics are redundant and that a reduced subset of metrics may be sufficient to measure the amount of demographic bias. The paper provides valuable insights for researchers in AI and related fields to mitigate dataset bias and improve the fairness and accuracy of AI models. The code is available at https://github.com/irisdominguez/dataset bias metrics.

show abstract

“…Some recent works have already found specific biases around gender, race, and age in both commercial FER systems [41], [42] and research models [43], [44], [45], [46], [47], [48]. From these works, Kim et al [41] focus on the age bias of commercial models in an age-labeled dataset.…”

Section: B Facial Expression Recognitionmentioning

confidence: 99%

“…From these works, Kim et al [41] focus on the age bias of commercial models in an age-labeled dataset. Ahmad et al [42] also study commercial models, but extend the research to age, gender, and race (considering two racial categories) by employing a custom database of politician videos. Regarding research models, Xu et al [44] study age, gender, and race bias in models trained on an Internet search-gathered dataset and evaluated on a different dataset with known demographic characteristics.…”

Section: B Facial Expression Recognitionmentioning

confidence: 99%

Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition

Dominguez-Catena¹,

Paternain²,

Galar³

2023

Preprint

0

View full text Add to dashboard Cite

Demographic biases in source datasets have been shown as one of the causes of unfairness and discrimination in the predictions of Machine Learning models. One of the most prominent types of demographic bias are statistical imbalances in the representation of demographic groups in the datasets. In this paper, we study the measurement of these biases by reviewing the existing metrics, including those that can be borrowed from other disciplines. We develop a taxonomy for the classification of these metrics, providing a practical guide for the selection of appropriate metrics. To illustrate the utility of our framework, and to further understand the practical characteristics of the metrics, we conduct a case study of 20 datasets used in Facial Emotion Recognition (FER), analyzing the biases present in them. Our experimental results show that many metrics are redundant and that a reduced subset of metrics may be sufficient to measure the amount of demographic bias. The paper provides valuable insights for researchers in AI and related fields to mitigate dataset bias and improve the fairness and accuracy of AI models. The code is available at https://github.com/irisdominguez/dataset bias metrics.

show abstract