Is Content-Related Evidence Useful in Validating Selection Tests?

Abstract

The 12 papers commenting on K. R. Murphy (2009a) raise a number of important issues, most of which can be subsumed in one of four themes. First, papers examining content‐oriented validation strategies are still necessary and useful, in part because of the frequent use of these strategies in the practice of industrial–organizational (I–O) psychology. Second, the term “content validity” means many different things both within and beyond the field of I–O psychology, and it is useful to understand what sorts of inferences examinations of test content do and do not support. Third, these 12 papers present very little evidence that content validation, as typically carried out by I–O psychologists, actually provides information about the likelihood that people who do well on the test will do well on the job. Finally, I believe that the best use of content‐related evidence in validating selection tests is in developing hypotheses about relationships between test scores and criteria rather than in testing these hypotheses.

Pervasiveness of Dominant General Factors in Organizational Measurement

“…Clearly, the labels used to represent content and content validity are not necessarily a reliable indicator of what is being measured (Murphy, 2009a(Murphy, , 2009b, nor are the labeled constructs valid indicators of the source of prediction of performance (Murphy, Deckart, Kinney, & Kung, 2013). When developing a new construct, few researchers look for a general factor (Drasgow, Nye, Carretta, Ree, 2010;Kyllonen, 1993;Stauffer, Ree, & Carretta, 1996).…”

Subject Matter Expert Judgments Regarding the Relative Importance of Competencies are not Useful for Choosing the Test Batteries that Best Predict Performance

“…Finally, we do not wish to suggest that comparisons between the content of tests and the content of jobs are irrelevant or worthless. In several articles (e.g., Murphy, 2009a, 2009b, 2009c; Murphy et al, ), we have argued that the extent to which selection tests match the competencies that are judged to be most important to the job is an important determinant of the acceptability of tests to test users. We have also noted that comparisons between the content of jobs and the content of test batteries are still recognized by the American courts and enforcement agencies as an appropriate strategy for demonstrating the validity of tests.…”

“…The results of literature reviews, coupled with analytic assessments of the validity of alternate selection test batteries chosen from a population of tests that show positive manifold, led Murphy and his colleagues (Murphy, 2009a, 2009b, 2009c; Murphy et al, ) to conclude that the match or mismatch between job content and test content had little relevance when determining the validity of selection test batteries as predictors of performance. That is, if one test battery measures competencies that are judged to be important to a job and another that measures competencies that are believed to be less important to the job, it is not likely that they will differ systematically in their criterion‐related validity.…”

“…In a series of recent articles, Murphy and his colleagues (e.g., Murphy, 2009a, 2009b; Murphy et al, ; Industrial Organizational Psychology: Perspectives on Science and Practice , vol. 2, no.…”

At Sea With Synthetic Validity

“…Murphy (2009a) gets around this problem by asserting that this is all we really need because most other predictors are derivable from these two (i.e., positive manifold). Later, Murphy (2009b) admits that this is more true for the cognitive than the personality domain, highlighting that there are still several relevant predictors that cannot be justified through validity generalization alone (Tett & Christiansen, 2007). Still, Murphy is quite right that the remaining predictors almost certainly will not be as important as GMA and Conscientiousness, and that we have encountered a law of diminishing returns.…”