Meta-Analysis in Organizational Research: A Guide to Methodological Options

Abstract

Meta-analysis provides a powerful tool for integrating findings from the research literature and building statistical models to explore trends and inconsistencies in the research base. Meta-analysis starts with a process for translating results from each study into an effect size that represents all findings in a common metric. Statistical models are then applied to estimate the mean, variance, and moderators of effect size. This article explores several key decision points in conducting a meta-analysis, including issues in obtaining a common metric, accounting for psychometric artifacts, and choosing an appropriate statistical model. It provides recommendations for choosing among alternate approaches and reporting results to ensure transparency. Expected final online publication date for the Annual Review of Organizational Psychology and Organizational Behavior, Volume 10 is January 2023. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Revisiting Sackett et al.’s (2022) rationale behind their recommendation against correcting for range restriction in concurrent validation studies.

“…Still, modest RR is not “no RR.” As any scientific subject, RR should be treated as a matter of degree rather than dichotomy. All of these thoughts suggest to us that Sackett et al’s recommendation against correcting for RR in concurrent validation studies would only push the field of personnel selection away from, rather than bring us closer to, the truth more than the slight under- or overcorrections for RR that may result from using good enough alternatives (e.g., using the national norm SD x available in test manuals with proper adjustment in lieu of the local applicant-based SD x ; Hoffman, 1995; also see Ones & Viswesvaran, 2003; Sackett & Ostgaard, 1994; a series of sensitivity analyses [RR corrections] using different u x values; Morris, 2023; also see Society for Industrial & Organizational Psychology [SIOP], 2018). That is, Sackett et al should (also) have recommended a good (enough) solution instead of pushing other selection researchers and practitioners (even themselves) to, in our view, an impossibly perfect solution to correcting for RR 16 and therefore ending up recommending no RR correction in concurrent validation studies.…”

“…In conclusion, all the evidence provided in this study suggests to us that Sacket et al’s recommendation against correcting for RR in estimating operational validity in concurrent validation studies should not be adopted. Moving forward, as noted in Morris (2023), “additional work is needed to fully understand the representativeness of range restriction estimates and optimal correction procedures under typical conditions” (i.e., concurrent validation studies; p. 20.14, parenthesis added).…”

“…However, other alternative, albeit less ideal, RR correction methods have been available and used in the field of personnel selection (e.g., Sackett & Ostgaard, 1994). Moreover, Morris (2023) states that "the best solution may be to apply multiple corrections based on different assumptions and to explicitly discuss the sensitivity of results to the choices regarding artifact values" (p. 20.15). This view is also aligned with the latest edition of the Principles for the Validation and Use of Personnel Selection Procedures (SIOP, 2018), which states that: "it is best to provide both corrected and uncorrected estimates of criterion-related validity" (p. 14).…”

“…First, they state that “[this approach] violates the assumption that the studies providing the artifact distribution constitute effectively a random draw of studies” (p. 2041). This is because such studies are mostly predictive studies, whereas VG meta-analyses generally include a large number of concurrent studies (Morris, 2023; Sackett et al, 1985). In other words, the distribution of the u x values constructed as mentioned above may suffer systematic sampling bias and thus lack credibility.…”

“…However, a different perspective exists. For example, Morris (2023) states, “while is it important to consider the representativeness of the data used for artifact corrections, it is not clear that predictive and concurrent designs will necessarily produce substantially different levels of range restriction” (p. 20.14; also see Schmidt et al, 1985 2 ). Second, Sackett et al state that “(range) restriction, if present, is indirect” and “little to no” in concurrent validation studies.…”