No abstract

The Schmidt‐Hunter interactive validity generalization procedure was applied to validity data for cognitive abilities tests for law enforcement occupations. Both assumed artifact distributions, and distributions of artifacts constructed from information contained in the current sample of studies were used to test the hypothesis of situational specificity and to estimate validity generalizability. Results for studies using a criterion of performance in training programs showed that validities ranged from .41 to .71, and for four test types the hypothesis of situational specificity could be rejected using the 75% decision rule. For the remaining test types, validity was generalizable, based on 90% credibility values ranging from .37 to .71. Results for studies using a criterion of performance on the job indicated that the hypothesis of situational specificity was not tenable for three test types, which had validities between .17 and .31. For the remaining test types, estimated mean true validities ranged from .10 to .26 and were generalizable to a majority of situations. Results for both groups of studies were essentially identical for the two types of artifact distribution. Possible reasons for the apparently lower validities and lesser generalizability for job performance criteria are discussed, including possible low validity of the criterion (due to lack of opportunity by supervisors to observe behavior) and the potential role of noncognitive factors in the determination of law enforcement job success. Suggestions for specifically targeted additional research are made.

This comment shows that the conclusion of Schmitt, Gooding, Noe, and Kirsch (1984) that their meta‐analytic findings are inconsistent with earlier validity generalization work is in error. The findings in their study that less variance than previously reported was due to sampling error are a result of their larger average sample sizes. Their claim that, after sampling error variance was accounted for, much unexplained variance remained, is incorrect. This error is demonstrated to be a result of their exclusive concentration on percentages and consequent failure to examine amount of observed and residual variance.

In a previous study, Schmidt, Hunter, Croll and McKenzie (1983) demonstrated that estimates of the validity of cognitive tests made by highly trained and experienced judges are more accurate than empirical estimates obtained from small‐sample validity studies. The present study examined whether less experienced judges could also produce accurate estimates. Twenty‐eight recent Ph.D.'s in I/O Psychology estimated observed validities for the same 54 job‐test combinations used by Schmidt et al. (1983). The estimates of these judges contained about twice as much random error as the experts' estimates. Systematic error of the less experienced judges was also greater than that of the experts (.0732 vs .019). The systematic errors of the two sets of judges were in opposite directions: less experienced judges overestimated validities, on average, while experts underestimated them. The results show that the estimates of less experienced judges contain less information than those of experts, but also that averages of estimates of several less experienced judges are as accurate as those obtained from small‐sample empirical studies.

No abstract