A Comparison of Criteria for Test Validation: A Meta‐analytic Investigation

Abstract

Meta‐analyses of validity coefficients from tests of clerical abilities for five criteria—supervisor ratings, supervisor rankings, work samples, production quantity, and production quality—were conducted, and the resulting expected true validities were compared. Ratings, rankings, work samples, and production quantity all resulted in high test validities. Validities resulting from ratings and quantity‐of‐production criteria were highly similar across tests. Validities resulting from rankings and work samples were on the average higher than those from ratings and quantity of production. The fifth criterion, quality of production, had low predictability and did not generalize across situations.

Using the Thorndike model to assess the fairness of cognitive ability tests for personnel selection

“…Unfortunately, they are also considered to be very narrow indices of job performance and, therefore, not as complete as supervisory ratings. This is a reasonable concern which is addressed by Nathan & Alexander (1988) and Hoffman, Nathan & Holden (1991) who found that subjective and objective measures were both predicted well by measures of cognitive ability, lending support for the equivalency of the two types of job performance measures.…”

“…For example, the standards of job performance, as measured by objective criteria, are based on a subjective decision (Nathan & Alexander, 1988). Thus, categorizing a measure as "objective" may hide the "subjective" decisions that went into formulating the measure.…”

A Criterion-Related Validation Study of the Army Core Leader Competency Model

“…Research has shown that objective and subjective criterion measures produce similar validities (Nathan & Alexander, 1988;Schmitt, Gooding, Noe, & Kirsch, 1984). In a metaanalysis of validity coefficients from tests of clerical workers for five criteria -supervisor ratings, supervisor rankings, work samples, production quantity and production quality, Nathan and Alexander (1988) found that the predictabilities of subjective ratings and objective production quantity were very similar.…”

Roadmap: An Agenda for Joint-Service Classification Research

“…Certainly there is a case to be made against ratings on the basis of their frequently observed low reliabilities and the high intercorrelations among different dimensions (i.e., halo). However, more recent evaluations have not been so negative (e.g., Campbell, McCloy, Oppler, & Sager, 1993;Nathan & Alexander, 1988). They suggest that ratings are a useful measurement method that deserve additional consideration.…”