Effects of range restriction and criterion contamination on differential validity of the SAT by race/ethnicity and sex.

Abstract

We illustrate the effects of range restriction and a form of criterion contamination (individual differences in course-taking patterns) on the validity of SAT scores for predicting college academic performance. College data facilitate exploration of differential validity’s determinants because they (a) permit the use multivariate range-restriction corrections to more accurately account for differential range restriction across subgroups and (b) allow for separate examinations of composite performance and specific performance episodes, the latter of which controls for ecological contamination of composite performance due to individuals’ choices of performance opportunities. Using data from 363,004 students at 107 U.S. institutions, we found that controlling for course-taking patterns resulted in validity coefficients that were appreciably larger than predictors’ correlations with obtained grade point averages (GPAs). The validities of SAT scores for predicting the first-year college performance of Black and Hispanic students were not significantly different from the validity for White students after correcting for both course-taking patterns and differential range restriction, but significant Black–White differences were detected for predicting 4-year cumulative performance. Validity estimates for predicting both first-year and 4-year cumulative performance were significantly smaller among Asian students than White students after making these corrections. The SAT’s observed validity for predicting college GPAs was substantially lower for males than females and, unexpectedly, controlling for course-taking patterns increased male-female validity differences. Implications for personnel selection research are discussed.

Bias, Fairness, and Validity in Graduate Admissions: A Psychometric Perspective

“…With respect to differential prediction, we were unable to find any large-scale research examining the potential for differential prediction using GRE data. There is, however, a large body of research examining and debating whether the SAT demonstrates differential prediction (e.g., Aguinis et al, 2010;Berry et al, 2011;Dahlke et al, 2019;Fischer et al, 2013;Mattern & Patterson, 2013). The general consensus from this research is that the SAT tends to overpredict undergraduate GPAs for Black students compared to White students and tends to underpredict undergraduate GPAs for female students compared to male students.…”

On the assessment of predictive bias in selection systems with multiple predictors.

“…However, this is more a limitation of the GPA metric in general than of our study in particular. Dahlke et al (2019) recently investigated whether criterion contamination in the form of individual differences in course-taking choices could explain White-minority differential validity of the SAT. They found that White–Black and White–Hispanic differential validity disappeared after controlling for differences in students’ course-taking choices.…”

“…These data were nonrandomly sampled by virtue of being collected from schools that used the SAT as a selection device and that consented to provide students’ grade data to the College Board for research purposes. The database we analyzed has been used in other published research (e.g., Beatty et al, 2015; Dahlke et al, 2018, 2019; Higdem et al, 2016; Kostal et al, 2015; Shewach et al, 2017; Yu et al, 2016), but the present study reports unique analyses of these data and draws novel insights.…”

Scale Mean and Variance Differences in MTurk and Non-MTurk Samples

“…This lack of research is surprising, given that Landers and Behrend’s (2015) discussion of convenience samples in organizational research emphasized range restriction as an inevitable issue across all convenience samples. Range restriction – when a sample’s variance is smaller than the variance in the target population – can limit generalizability, resulting in weaker correlations between variables of interest (Dahlke et al, 2019; McAbee & Oswald, 2017; Sackett & Yang, 2000).…”