In Pursuit of Interpretable, Fair and Accurate Machine Learning for Criminal Recidivism Prediction

Abstract

Our goal is to study the predictive performance, interpretability, and fairness of machine learning models for pretrial recidivism prediction. Machine learning methods are known for their ability to automatically generate high-performance models (that sometimes even surpass human performance) from data alone. However, many of the most common machine learning approaches produce "black-box" models-models that perform well, but are too complicated for humans to understand. "Interpretable" machine learning techniques seek to produce the best of both worlds: models that perform as well as black-box approaches, but also are understandable to humans. In this study, we generate multiple black-box and interpretable machine learning models. We compare the predictive performance and fairness of the machine learning models we generate, against two models that are currently used in the justice system to predict pretrial recidivism-namely, the Risk of General Recidivism and Risk of Violent Recidivism scores from the COMPAS suite, and the New Criminal Activity and New Violent Criminal Activity scores from the Arnold Public Safety Assessment.We first evaluate the predictive performance of all models, based on their ability to predict recidivism for six different types of crime (general, violent, drug, property, felony, and misdemeanor). Recidivism is defined as a new charge for which an individual is convicted within a specified time frame (which we specify as six months or two years). We consider each type of recidivism over the two time periods to control for time, rather than to consider predictions over an arbitrarily long or short pretrial period. Next, we examine whether a model constructed using data from one region suffers in predictive performance when applied to predict recidivism in another region. Finally, we consider the latest fairness definitions created by the machine learning community. Using these definitions, we examine the behavior of the interpretable models, COMPAS, and the Arnold Public Safety Assessment, on race and gender subgroups.Our findings and contributions can be summarized as follows:• We contribute a set of interpretable machine learning models that can predict recidivism as well as black-box machine learning methods and better than COMPAS or the Arnold Public Safety Assessment for the location they were designed for. These models are potentially useful in practice. Similar to the Arnold Public Safety Assessment, some of these interpretable models can be written down as a simple table that fits on one page of paper. Others can be displayed using a set of visualizations.• We find that recidivism prediction models that are constructed using data from one location do not tend to perform as well when they are used to predict recidivism in another location, leading us to conclude that models should be constructed on data from the location where they are meant to be used, and updated periodically over time.• We reviewed the recent literature on algorithmic fairness, but most of the fairness criteria do...

EXMOS: Explanatory Model Steering through Multifaceted Explanations and Data Configurations

“…Over the past decade, the field of XAI has witnessed numerous studies being conducted to measure the efficacy of various explanation methods for increasing the transparency of ML systems across multiple application domains, such as healthcare [8,16,18,60], finance [9,13,24], and law enforcement [72,79,87]. Along with making black-box ML models more transparent, XAI methods have also aimed to make these systems more understandable and trustworthy [8,49,54].…”

The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations

“…Post-processing solutions to standard explanation model training could also prove effective, similar to recent work in the space of improving worst-case generalization [76]. However, such solutions need to be appraised carefully to ensure that the resulting models are both fair and remain interpretable to users [110].…”

Towards Directive Explanations: Crafting Explainable AI Systems for Actionable Human-AI Interactions

“…The utilisation of Artificial Intelligence (AI) systems has grown significantly in the past few years across diverse domains such as medical [13,14,37], finance [10,12,15], legal [44,51,54] and others [2,6,36]. Despite the success of AI systems across various applications, the "black-box" nature of AI models has raised several concerns related to lack of transparency [6,29,34] and appropriate trust [27,47], particularly when predicted outcomes are biased, unfair, incorrect or misguiding [11,22,33].…”