Differential Validity and Differential Prediction of Cognitive Ability Tests: Understanding Test Bias in the Employment Context

Abstract

Substantial mean score differences and significant adverse impact have long motivated the question of whether cognitive ability tests are biased against certain non-White subgroups. This article presents a framework for understanding the interrelated issues of adverse impact and test bias, with particular focus on two forms of test bias especially relevant for personnel selection: differential validity and differential prediction. Ethical and legal reasons that organizations should be concerned about differential validity/prediction are discussed. This article also serves as a critical review of the research literature on differential validity/prediction. The general conclusion is that available evidence supports the existence of differential validity/ prediction in the form of correlation/slope and intercept differences between White and non-White subgroups. Implications for individuals and organizations are outlined, and a future research agenda is proposed highlighting the need for new, better data; new, better methods of testing for differential validity/prediction; and investigation of substantive factors causing differential validity/prediction.

Why Do Achievement Measures Underpredict Female Academic Performance?

“…A good deal of research has been conducted evaluating whether measures used to select applicants—whether for admittance to a college, selection for a job, or enlistment into the military—result in differential prediction of future performance for various subgroups (Aguinis, Culpepper, & Pierce, ; Berry, ; Berry & Zhao, ; Mattern & Patterson, ). In the context of college admission decisions, an extensive body of literature indicates that admission test scores and high school grade point average (HSGPA) overpredict college performance for underrepresented minority students and underpredict college performance for females (Burton & Ramist, ; Mattern & Patterson, ; Mattern, Patterson, Shaw, Kobrin, & Barbuti, ; Radunzel & Noble, ; Sanchez, ; Young, ).…”

Subgroup differences in situational judgment test scores: Evidence from large applicant samples

“…Fifth, our analyses focus on subgroup differences in SJTs and do not shed a light on other aspects that influence bias against subgroups in selection (see, for overviews, Berry, ; Hough et al, ; Ployhart & Holtz, ). For example, we could not investigate whether the SJTs in our study show substantially different correlations to outcomes such as job performance for different ethnic or sex groups because criterion data were unavailable.…”

Gender‐Based Differential Prediction by Curriculum Samples for College Admissions

“…Tests for slope differences tend to be underpowered, even in large samples, and tests for intercept differences tend to have inflated Type I errors (Aguinis et al., ). There have been suggestions to overcome these problems (Aguinis et al., ; Berry, ; Mattern & Patterson, ; Meade & Fetzer, ), but most suggestions rely on visual inspection, or assume that there are no slope differences, or that they are difficult to implement (for example, improving test reliability and reducing subgroup score differences; e.g., Berry, ). A Bayesian approach does not solve all these problems, but inconclusive results can be distinguished from evidence in favor of the null hypothesis of no differential prediction.…”

“…There were several reasons to supplement the classical frequentist analyses with a Bayesian approach (e.g., Gelman et al., ; Kruschke et al., ). First, there are some shortcomings of the classical step‐down regression analysis (Lautenschlager & Mendoza, ) to study differential prediction (Aguinis et al., ; Berry, ; Meade & Fetzer, ). Tests for slope differences tend to be underpowered, even in large samples, and tests for intercept differences tend to have inflated Type I errors (Aguinis et al., ).…”