Revisiting Sackett et al.’s (2022) rationale behind their recommendation against correcting for range restriction in concurrent validation studies.

Abstract

Sackett et al. (2022) recommend against correcting for range restriction (RR) in concurrent validation studies. The main rationale behind their recommendation is that unless "r zx " (an unrestricted true-score correlation between the third variable Z where actual selection occurred in a top-down manner [a.k.a., suitability] and the predictor of interest, X) is as high as .90 and selection ratios are as low as .30-both unlikely events in their view, the degree of RR (u x ) in concurrent validation studies is unlikely to be low enough (i.e., lower than .90) to warrant RR correction. That is, (a) the "r zx " ≥ .90 and (b) the selection ratio ≤ .30 are two critical conditions for the third condition, (c) u x ≤ .90, a need for RR correction. In this study, we revisit each of these conditions that constitute the rationale behind their recommendation: (a) whether "r zx " is unlikely to be as high as .90; (b) whether selection ratios of .30 or lower are "extreme"; and (c) whether the degree of RR is "little to no" (i.e., u x ≥ .90) in concurrent validation studies, thus no need for correcting for RR in concurrent validation studies. First, our reanalysis of their Table 1 indicates that it is not implausible that "r zx " is as high as .90. Second, several studies report that selection ratios of .30 or lower are not extreme. Finally, our reanalysis of their Table 5 indicates that Sackett et al. substantially underestimate the severity of RR and its biasing effect on operational validity in concurrent validation studies due to their use of a particular RR correction method (Case IV). We believe these findings suggest that there is not sufficient support for the rationale behind Sackett et al.'s recommendation and, thus, their recommendation itself should be reconsidered.

Correcting for range restriction in meta-analysis: A reply to Oh et al. (2023).

“…Although it is not related to Oh et al's (2023) suggestion that we did not define r zx as an unrestricted correlation, in a friendly review, a colleague notes one minor area in which we were inconsistent in our use of "r zx ." In the first two panels of Table 1 in Sackett et al (2022), we presented findings with no measurement error.…”

“…Validation Studies Oh et al (2023) attribute to us the position that one should not correct for range restriction in concurrent validity studies. This permeates their article and is even found in the title: "Revisiting Sackett et al's (2022) Rationale Behind Their Recommendation Against Correcting for Range Restriction in Concurrent Validation Studies."…”

“…Oh et al note that we do not offer citations to support our inferences about the frequency of these various combinations of r zx and selection ratios but later acknowledge that this information is lacking in the field. Oh et al's (2023) case requires that the bulk of the concurrent validation research contributing to meta-analyses comes either from (a) settings in which one does a concurrent study on a predictor with an extremely high correlation with the original basis for selection (r zx ) or from (b) settings with strong (i.e., greater than .50) correlations between z and x paired with a small selection ratio (i.e., .10 or smaller). Their case hinges on this because Sackett et al (2022) showed that those are the only scenarios in which indirect range restriction can have a sizable effect on criterion-related validity.…”

“…On article page 1304, Oh et al (2023) wrote, "Sackett et al ( 2022) state that selection ratios of .10-.30 are extremely low." We reviewed Sackett et al and found 12 instances of the use of the word "extreme."…”

“…We offered revised estimates of operational validity (i.e., the level of validity expected after correcting for unreliability in the criterion and for range restriction, where appropriate), which are often quite different from the prior estimates. Oh et al (2023) replied to our article, offering a different perspective on a number of choices we made and conclusions we drew. Here we respond to central themes in their reply.…”