Corrections for Criterion Reliability in Validity Generalization: A False Prophet in a Land of Suspended Judgment

Abstract

The results of meta-analytic (MA) and validity generalization (VG) studies continue to be impressive. In contrast to earlier findings that capped the variance accounted for in job performance at roughly 16%, many recent studies suggest that a single predictor variable can account for between 16 and 36% of the variance in some aspect of job performance. This article argues that this “enhancement” in variance accounted for is often attributable not to improvements in science but to a dumbing down of the standards for the values of statistics used in correction equations. With rare exceptions, applied researchers have suspended judgment about what is and is not an acceptable threshold for criterion reliability in their quest for higher validities. We demonstrate a statistical dysfunction that is a direct result of using low criterion reliabilities in corrections for attenuation. Corrections typically applied to a single predictor in a VG study are instead applied to multiple predictors. A multiple correlation analysis is then conducted on corrected validity coefficients. It is shown that the corrections often used in single predictor studies yield a squared multiple correlation that appears suspect. Basically, the multiple predictor study exposes the tenuous statistical foundation of using abjectly low criterion reliabilities in single predictor VG studies. Recommendations for restoring scientific integrity to the meta-analyses that permeate industrial–organizational (I–O) psychology are offered.

Frequency versus time lost measures of absenteeism: Is the voluntariness distinction an urban legend?

“…Also, we concur with Charles () that a degree of imprecision in correction is better than assuming error‐free measurement. For more general cautions concerning corrections applied under conditions of low criterion reliability, consult Le Breton, Scherer, and James ().…”

“…Also, we concur with Charles (2005) that a degree of imprecision in correction is better than assuming error-free measurement. For more general cautions concerning corrections applied under conditions of low criterion reliability, consult Le Breton, Scherer, and James (2014). Note: K = number of independent samples; N = total number of individuals; r = sample size-weighted mean of correlations; ρ = estimated true score correlation; SD ρ = standard deviation of ρ; σ 2 ρ = variance in the population correlations; % variance = percent of variance accounted for by sampling and measurement error; CV 80 = 80% credibility interval; CI 95 = 95% confidence interval.…”

Is a .51 validity coefficient good? Value sensitivity for interview validity

“…Although expressing a validity coefficient in terms of explained variance allows for direct interpretation of magnitude, assumptions about how much variance should be explained poses problems for even sophisticated consumers of the information. The idea that it is theoretically possible to explain 100% of the variance in job performance at the time of hire is absurd, and neglects not only the many other determinants of performance (LeBreton, Scherer, & James, ) but also the uncertainty that is inherent in nature (Salsburg, ). One possible solution to this problem is to survey selection experts on how much variance in job performance is theoretically explainable at the time of hire.…”

Untitled

“…Finally, our approach assumes that corrections for range restriction and criterion unreliability are appropriate. There indeed is some debate on this topic (LeBreton, Scherer, & James, 2014;Shen, Cucina, Walmsley, & Seltzer, 2014;Viswesvaran, Ones, Schmidt, Le, & Oh, 2014). We leave it to readers to decide where they stand on this debate and whether or not to make corrections and which values to apply when using corrections.…”