Multiplicative validity generalization model: Accuracy of estimates as a function of sample size and mean, variance, and shape of distribution of true validities.

Abstract

The accuracy of estimates of the mean, variance, and lower credibility value of true validities produced by the independent multiplicative model (Callender & Osburn, 1980) and by a modified dependent model that takes the correlation (if any) between range restriction and criterion reliability artifacts into account were studied using simulation techniques. Sample sizes (w = 50 or 100) and the number of studies per analysis (50) were selected to be consistent with the typical parameters found in 129 published validity generalization analyses. The mean (.25 or .5), standard deviation (0. .11, or .18), and shape (normal, platykurtic, positive skew, or negative skew) of the true validity distributions were systematically varied. It was concluded that the independent and modified dependent estimates were typically accurate, although the credibility estimates were affected by the extremely skewed distributions. The mean and variance estimates from both models were not affected by the distribution shape. Overall, the results supported the applicability of our models and the Schmidt-Hunter noninteractive model. Some limitations of the "bare-bones" sampling-error-only approach were noted.

The Effect of Correlated Artifacts and True Validity on the Accuracy of Parameter Estimation in Validity Generalization

“…Because each of the artifact and true validity distributions consisted of 100 values, 100 separate validity studies were included in each VG study. According to Callender, Osburn, Greener, Ashworth (1982), 33% of the 129 published VG studies had a mean sample size above 100. This process was repeated 1,000 times, thus resulting in 1,000 meta-analyses of 100 validity studies with 100 subjects each.…”

Forty Questions About Validity Generalization and Meta‐analysis

“…We later dropped the Fisher's z transformation for two reasons. First, Callender, Osburn, Greener, and Ashworth (1982) had shown via simulation studies that the standard formula for the sampling error of the correlation coefficient was quite accurate despite use of the observed validity in this formula. (This study was made available to us in 1979 in prepublication form).…”

A note on validity generalization procedures.

“…An example presented shortly indicates a small overall bias in favor of a finding of cross-situational consistency for the illustrative data in Table 1 nonindependence between the pi and ei results in minor underestimation of the value of c 2 (Burke, 1984;Linn & Dunbar, 1982). Callender, Osburn, Greener, and Ashworth (1982) Schmidt and Hunter (1977) originally used zs in VG analysis to ensure against covariation between the pi and ei, but switched to observed correlations under the assumption that their formula for sampling error was "very accurate" (Schmidt et al, 1980, p. 660). Later, the reason for the switch from zs to rs was given as "the effect of Fisher's z transformation is to assign extra weight to large observed validity coefficients" (Schmidt et al, 1982, p. 839).…”