Accuracy of Population Validity and Cross-Validity Estimation: An Empirical Comparison of Formula-Based, Traditional Empirical, and Equal Weights Procedures

Abstract

An empirical monte carlo study was performed using predictor and criterion data from 84,808 U.S. Air Force enlistees. 501 samples were drawn for each of seven sample size conditions: 25, 40, 60, 80, 100, 150, and 200. Using an eight-predictor model, 500 estimates for each of 9 validity and 11 cross-validity estimation procedures were generated for each sample size condition. These estimates were then compared to the actual squared population validity and cross-validity in terms of mean bias and mean squared bias. For the regression models determined using ordinary least squares, the Ezekiel procedure produced the most accurate estimates of squared population validity (followed by the Smith and the Wherry procedures), and Burket’s formula resulted in the best estimates of squared population cross-validity. Other analyses compared the coefficients determined by traditional empirical cross-validation and equal weights; equal weights resulted in no loss of predictive accuracy and less shrinkage. Numerous issues for future basic research on validation and cross-validation are identified.

Healthcare systems and performance evaluation: comparison of performance indicators in v4 countries using models of composite indicators

“…In the case of a large number of indicators, it is diffi cult to adequately determine the weights for each indicator, and the more complex weighing scheme would make it diffi cult to interpret the impact of the individual sub-indicators. Similarly, if we have only a small set of data that consists of less than 30 observations, the same weights are appropriate (Raju et al, 1999). Hopkins (1991) says that if it is impossible to obtain a general consensus for weight determination, the simplest solution is the best.…”

“…In the event that we only have a small set of data, Raju et al (1999) chooses to use the same weights. Hopkins (1991) also claims that if it is impossible to gain common agreement for determining weights, the simplest solution is the best solution, i.e.…”

The Game Transfer Phenomena Scale: An Instrument for Investigating the Nonvolitional Effects of Video Game Playing

“…In regards to participants' gaming-related habits and behaviors, the majority were 'hardcore' players (65% in S1 and 55.8% in S2), played videogames mostly 'between 3 to 6 hours' (41.2% S1 and 43.2% S2) and reported a weekly gaming frequency of '2 to 4 days a week' (42.6% in S1 and 28.3% in S2). However, in S2, 40.3% (n = 265) reported playing videogames 'everyday' (see Table 1). …”

Sample Size for Multiple Regression: Obtaining Regression Coefficients That Are Accurate, Not Simply Significant.

“…Since B. F. Green (1977) showed that "many linear composites [that is, predicted scores] are barely different from using equal weights" (p. 274), the exchangeable structure offers a potentially useful tool when planning necessary sample size (see Maxwell, 2000, for a thorough treatment and rationale of the exchangeable structure, as well as a similar correlational structure that is somewhat relaxed). Many times an exchangeable structure may be a sensible place to start when planning sample size for a multiple regression analysis, unless there are obvious theoretical reasons not to do so (B. F. Green, 1977;Raju, Bilgic, Edwards, & Fleer, 1999;Wainer, 1976).…”