It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks

Abstract

Risk assessment instrument (RAI) datasets, particularly ProPublica's COMPAS dataset, are commonly used in algorithmic fairness papers due to benchmarking practices of comparing algorithms on datasets used in prior work. In many cases, this data is used as a benchmark to demonstrate good performance without accounting for the complexities of criminal justice (CJ) processes. We show that pretrial RAI datasets contain numerous measurement biases and errors inherent to CJ pretrial evidence and due to disparities in discretion and deployment, are limited in making claims about real-world outcomes, making the datasets a poor fit for benchmarking under assumptions of ground truth and real-world impact. Conventional practices of simply replicating previous data experiments may implicitly inherit or edify normative positions without explicitly interrogating assumptions. With context of how interdisciplinary fields have engaged in CJ research, algorithmic fairness practices are misaligned for meaningful contribution in the context of CJ, and would benefit from transparent engagement with normative considerations and values related to fairness, justice, and equality. These factors prompt questions about whether benchmarks for intrinsically socio-technical systems like the CJ system can exist in a beneficial and ethical way.

Algorithm Fairness in AI for Medicine and Healthcare

“…Indeed, it is now accepted that race is a social construct and that there is greater genetic variability within a particular race than there is between races [157][158][159] . As such, categorization of patients by race can obscure a host of potential confounders to fairness analyses including culture, history, and socioeconomic status that all may separately and synergistically influence a particular patient's health 160,161 . These manifold factors can also vary by location so that the same person may be considered of different races in different geographic locations, as seen in the example of self-reported Asian ethnicity in the TCGA versus Pioneer and self-reported race in COMPAS 55,161 .…”

Predictability and Surprise in Large Generative Models

“…We refer to the combination of these two properties throughout the paper as smooth general capability (or performance) scaling. 6 In Section 2.2, we discuss abrupt specific capability scaling, in which models can also suddenly gain specific capabilities at scale. We illustrate this phenomenon with three examples from the literature [11,56,4].…”

Cascaded Debiasing : Studying the Cumulative Effect of Multiple Fairness-Enhancing Interventions

“…As far as datasets go, we have conducted experiments on 4 datasets that have been used extensively in the fairness literature to benchmark different fairness enhancing interventions. Recent research efforts have questioned the validity of some these datasets like the COMPAS [3] and Adult Income dataset [12]. Future work might include more recent datasets [12] and other stages of intervention, such as the data curation stage into the analysis [22].…”