Correcting for range restriction in meta-analysis: A reply to Oh et al. (2023).

Abstract

Oh et al. (2023) question a number of choices made in our article (Sackett et al., 2022); here we respond. They interpret our article as recommending against correcting for range restriction in general in concurrent validation studies; yet, we emphasize that we endorse correction when one has access to the information needed to do so. Our focus was on making range restriction corrections when conducting meta-analyses, where it is common for primary studies to be silent as to the prior basis for selection of the employees later participating in the concurrent validation study. As such, the applicant pool information needed for correction is typically not available. Sackett et al. (2022) highlighted that in many situations, range restriction will be small; so, the inability to correct for it results in only a modest underestimate of validity. Oh et al. mention settings that would result in substantial range restriction; here, we present our rationale as to why we view such settings as uncommon rather than as making up the bulk of the studies contributing to meta-analyses.

A contemporary look at the relationship between general cognitive ability and job performance.

“…Sackett et al’s (2022) position has been challenged by Oh et al (in press), who argue that conditions under which concurrent studies produce large range restriction are more common than Sackett et al posit. Sackett et al (2023) agree that it is possible to obtain sizable range restriction in concurrent studies but argue that is not plausible that these conditions make up the bulk of the cumulative meta-analytic data on validity. Combining Zhou et al’s (2022) revised estimates of interrater reliability and Sackett et al’s reassessment of range restriction results in a mean operational validity estimate for GCA of .31, considerably smaller than Schmidt and Hunter’s (1998) mean value of .51.…”

A longitudinal meta-analysis of range restriction estimates and general mental ability validity coefficients: Better addressing overcorrection amid decline effects.

“…For the latter, the comparison group or nonrestricted SD is simply the Wave-specific occupational SD ratio multiplied by the national norm, which in turn is used to calculate the u x needed for range restriction corrections. Notably, our results indicate that Sackett et al (2023) were correct in that on average range restriction/ enhancement corrections are now largely superfluous (i.e., Hypothesis 4b), but that Oh et al (2023) were correct in that there remains many individual or applied cases where they are needed. Variations in heterogeneity, both up and down, are bound to occur due to labor market conditions, recruiting efforts, organizational attractiveness, layoffs, promotions, or simply sampling error.…”

“…In addition, there is a recent related major debate, comprising a focal article by Sackett et al (2022), a critical reply by Oh et al (2023), and followed by a rejoinder by Sackett et al (2023). Sackett et al's main thesis is that we should not correct for range restriction in concurrent studies if they lack the requisite information, arguing that this will generally not lead to substantial underestimates of validity.…”

Reducing subgroup differences in personnel selection through the application of machine learning

“…Examples include expanding the predictor space—to explore new predictors that contribute to optimizing multiple organizational objectives. Existing methodologies (e.g., regression) that can only analyze one outcome at a time have restricted the historical focus in personnel selection to predictors that have high correlation with task performance, such as cognitive ability, structured interviews, and biodata (Sackett et al., 2021). The vast majority of our understanding of personnel selection predictors are informed by meta‐analyses, regression, and structural equation modeling studies that examine the criterion‐related validity, composite validity, and/or incremental validity of predictor(s) in predicting a single workplace outcome—for example, job performance or retention.…”