Imperfect Corrections or Correct Imperfections? Psychometric Corrections in Meta-Analysis

Abstract

There is understandable concern by LeBreton, Scherer, and James (2014) that psychometric corrections in organizational research are nothing more than a form of statistical hydraulics. Statistical corrections for measurement error variance and range restriction might inappropriately ratchet observed effects upward into regions of practical significance and publication glory-at the expense of highly questionable results.We share this concern. Of course, effect sizes based on high-quality measurement and representative sampling from populations of interest are preferred over psychometric corrections applied to effects subject to measurement error variance and range restriction. But there are at least two major reasons to consider the latter. First and foremost, we must contend with the reality of our research: Measures are never perfect reflections of their intended constructs, and a variety of range restriction effects exist with respect to the population to which inferences are made (e.g., effects of recruiting, self-selection, and other nonrandom sampling). Statistically modeling measurement error variance and range restriction attempts to take these realities into account when estimating latent relationships.Second, assuming that the psychometric model and corrections are realistic (a strong assumption that we revisit shortly), the resulting magnitudes and patterns of latent relationships might be more accurate and interpretable than the uncorrected effects observed in the data. But you don't get something for nothing in making these psychometric corrections: Statistically,

Is the Deliberate Practice View Defensible? A Review of Evidence and Discussion of Issues

“…After their correction for measurement error variance (unreliability), the average correlation for the 14 studies that measured either purposeful practice or deliberate practice with performance increased from r = 0.54 to r = 0.78, indicating that purposeful/deliberate practice 4 explained 61% of the reliable variance in the performance. Although psychometric corrections come with larger standard errors and confidence intervals ( Oswald et al, 2015 ), this point estimate of 61% might be interpreted to support Ericsson et al’s (1993) central claim that individual differences in domain-relevant performance can largely be accounted for by accumulated amount of practice—at least a weak version of this claim (see Hambrick et al, 2018a ).…”

Obtaining Unbiased Results in Meta-Analysis: The Importance of Correcting for Statistical Artifacts

“…Applying statistical corrections increases the sampling error in the corrected effect sizes, leading to larger standard errors and wider confidence intervals. It is always better to reduce measurement error by using more reliable measurement procedures during data collection, rather than to rely on statistical corrections after the fact (Oswald, Ercan, McAbee, Ock, & Shaw, 2015).…”

“Where’s the I-O?” Artificial Intelligence and Machine Learning in Talent Management Systems

“…AI/ML refers to "artificial intelligence/machine learning". Potential Organizational Benefits and Limitations of Artificial Intelligence/Machine Learning correlations for measurement error cannot improve actual real-world predictions at the individual level (Oswald, Ercan, McAbee, & Shaw, 2015). Perhaps future AI/ML-enhanced technologies will aid in gathering better criterion data that, in turn, improve the performance of AI/ML-based tests and algorithms.…”