Statistical learning methods and cross-cultural fairness: Trade-offs and implications for risk assessment instruments.

Abstract

The use of statistical learning methods has recently increased within the risk assessment literature. They have primarily been used to increase accuracy and the area under the curve (AUC, i.e., discrimination). Processing approaches applied to statistical learning methods have also emerged to increase cross-cultural fairness. However, these approaches are rarely trialed in the forensic psychology discipline nor have they been trialed as an approach to increase fairness in Australia. The study included 380 Aboriginal and Torres Strait Islander and non-Aboriginal and Torres Strait Islander males assessed with the Level of Service/Risk Needs Responsivity (LS/RNR). Discrimination was assessed through the AUC, and fairness was assessed through the cross area under the curve (xAUC), error rate balance, calibration, predictive parity, and statistical parity. Logistic regression, penalized logistic regression, random forest, stochastic gradient boosting, and support vector machine algorithms using the LS/RNR risk factors were used to compare performance against the LS/RNR total risk score. The algorithms were then subjected to pre- and postprocessing approaches to see if fairness could be improved. Statistical learning methods were found to produce comparable or marginally improved AUC values. Processing approaches increased several fairness definitions (namely xAUC, error rate balance, and statistical parity) between Aboriginal and Torres Strait Islanders and non-Aboriginal and Torres Strait Islanders. The findings demonstrate that statistical learning methods may be a useful approach to increasing the discrimination and cross-cultural fairness of risk assessment instruments. However, both fairness and the use of statistical learning methods encompass significant trade-offs that need to be considered.

Testing Predictive Biases at the Intersection of Race-Ethnicity and Sex: A Multi-Site Evaluation of a Pretrial Risk Assessment Tool

“…cUrrenT sTUdy we use the approach detailed in the Standards and define instrument bias as differential prediction if regression lines differ between racial-ethnic and sex groups. There are several ways to assess bias with actuarial instruments that include error rate balance (e.g., equal false positives and false negatives across groups), predictive parity (e.g., equal outcome rates by groups at specific cutoffs), and calibration (e.g., equal probabilities across scores for groups; Ashford et al, 2023). It is difficult to find parity across all measures of bias simultaneously unless there are equal base rates in outcomes or perfect prediction (Chouldechova, 2017).…”