The Effect of Correlated Artifacts and True Validity on the Accuracy of Parameter Estimation in Validity Generalization

Abstract

The effect of non-zero intercorrelations among the three artifacts (criterion reliability, predictor reliability, and range restriction on the predictor) and true validity on the accuracy of estimation in two validity generalization models (Model 1 and Model 2) was investigated. Only the Thylor Series Approximation 1 (TSA1) procedure from Model 1 and the procedure with complete sample-based artifact data from Model 2 were included in this study. Six intercorrelation conditions (zero, low positive, medium positive, low negative, low negative and positive, and medium negative and positive) and two distributions of artifacts were used in this investigation. The two major conclusions from this study are: (a) Both models yielded reasonably accurate estimates of the mean of true validities (M,) and (b) only Model 2 produced accurate estimates of the variance of true validities (V,). The need for additional research for accurately estimating V, when the three artifacts and true validity are correlated is recommended. The generalizability of an empirical result is a critical issue in all scientific research. In the field of industrial and organizational psychology, one generalizability issue, namely, predictor-criterion relationships in personnel selection, has been the focus of much research over the last 15 years. In their seminal paper, Schmidt and Hunter (1977) proposed a model and a procedure for assessing the generalizability of validity across populations. Their model for validity generalization (VG) may be expressed as Portions of this paper were previously presented at the 1995 Academy of Management The authors express their appreciation to three anonymous reviewers for their helpful Correspondence and requests for reprints should be addressed to Nambury S. Raju, Annual Meeting Vancouver, BC. comments. Institute of Psychology, Illinois Institute of Technology, Chicago, IL 60616-3793. COPYRIGHT 0 1990 PERSONNEL PSYCHOLOGY, INC 453 454 PERSONNEL PSYCHOLOGY U rxy = p l y 6 6 dl + (.2 -qp$pxxpgy + e l PIwhere T ,~ is the observed correlation between the predictor (2) and criterion (y), pxy is the unattenuated and unrestricted population correlation between 2 and y (also referred to as the true validity), pxx and pyy are the unrestricted population reliabilities of 2 and y, respectively, u is the ratio of restricted population standard deviation to unrestricted population standard deviation on the predictor, and e is the sampling error. A major goal of VG analysis is the estimation of the mean (M,) and variance (V,) of true validities across populations. Five different procedures are currently available for estimating M, and V,. These procedures, based on the VG model (Model 1) given in Equation 1 above, are the interactive procedure (Pearlman, Schmidt,

An Evaluation of James et al.'s (1992) VG Estimation Procedure when Artifacts and True Validity are Correlated*

“…Assumption 3). Raju et al (1998) showed that even if correlations do exist among the artifacts and r across studies, the accuracy of the estimation procedure derived from Model 2 was not affected.…”

“…James et al's descriptions of the hypothesized effects of situational moderators on the accuracy and/or the appropriateness of the current VG procedures are analytical in nature and thus require empirical verification. In a recent Monte Carlo study, Raju, Anselmi, Goodman, and Thomas (1998) have shown that if the third assumption is violated, then the estimates of the variance of r's derived from a procedure (TSA1) based on artifact distributions are not as accurate as the variance estimates based on a procedure that does not require this assumption; with respect to estimating the mean of r's, the TSA1 procedure performs better than the other procedure. It is important to note that in the Raju et al study, the violation of Assumption 3 that resulted in inaccurate estimates of V r was a correlation between r xy and the artifacts.…”

“…According to Mendoza and Mumford (1987), such direct range restriction does not violate the assumption of uncorrelated true and error scores. In the previously reported Monte Carlo studies, Raju et al (1991Raju et al ( , 1998 stated that the direct range restriction on the predictor was on the observed score and hence the estimate of predictor reliability (defined as the ratio of true score variance to observed score variance) from the restricted sample may have overestimated the reliability defined as the correlation between two parallel forms (Mendoza & Mumford, 1987, p. 290). According to Mendoza and Mumford, the parallel form reliability is the correct definition of reliability in the restricted sample when the range restriction is direct on the observed score.…”

“…When this happens, Raju et al's Model 2 procedure also requires Assumption 2 and its estimates are still quite accurate according to Raju et al (1991) and may be less accurate according to Law, Schmidt, and Hunter (1994b). While the Model 1-based procedures may not require sample-based data, their accuracy can be a problem if Assumptions 2 and 3 are not satisfied (Raju et al, 1986(Raju et al, , 1989(Raju et al, , 1991(Raju et al, , 1998.…”

Test development and use: New twists on old questions

“…The most important of these reasons is sampling error. The many VG studies and technical refinements of VG procedures that now exist in the literature in applied psychology (e.g., Aguinis, 2001;Aguinis & Whitehead, 1997;Hunter & Schmidt, 2004;Raju, Anselmi, Goodman, & Thomas, 1998;Schmidt & Hunter, 1998) suggest that VG is a robust phenomenon. Perhaps the most important implication of this work is that it has called attention to the fact that the mean of several validity coefficients may be a better …over the past several decades, there have been some significant advances in psychological science, specifically in our knowledge about important questions to address with respect to the development and use of assessment tools.…”