Improving our understanding of predictive bias in testing.

Abstract

Predictive bias (i.e., differential prediction) means that regression equations predicting performance differ across groups based on protected status (e.g., ethnicity, sexual orientation, sexual identity, pregnancy, disability, and religion). Thus, making prescreening, admissions, and selection decisions when predictive bias exists violates principles of fairness based on equal treatment and opportunity. First, we conducted a two-part study showing that different types of predictive bias exist. Specifically, we conducted a Monte Carlo simulation showing that out-of-sample predictions provide a more precise understanding of the nature of predictive bias—whether it is based on intercept and/or slope differences across groups. Then, we conducted a college admissions study based on 29,734 Black and 304,372 White students, and 35,681 Latinx and 308,818 White students and provided evidence about the existence of both intercept- and slope-based predictive bias. Third, we discuss the nature and different types of predictive bias and offer analytical work to explain why each type exists, thereby providing insights into the causes of different types of predictive bias. We also map the statistical causes of predictive bias onto the existing literature on likely underlying psychological and contextual mechanisms. Overall, we hope our article will help reorient future predictive bias research from whether it exists to the why of different types of predictive bias.