A Meta‐analysis of Work Sample Test Validity: Updating and Integrating Some Classic Literature

Abstract

Work sample tests have been used in applied psychology for decades as important predictors of job performance, and they have been suggested to be among the most valid predictors of job performance. As we examined classic work sample literature, we found the narrative review by Asher and Sciarrino (1974) to be plagued by many methodological problems. Further, it is possible that data used in this study may have influenced the results (e.g., r = .54) reported by Hunter and Hunter in their seminal work in 1984. After integrating all of the relevant data, we found an observed mean correlation between work sample tests and measures of job performance of .26. This value increased to .33 when measures of job performance (e.g., supervisory ratings) were corrected for attenuation. Our results suggest that the level of the validity for work sample tests may not be as large as previously thought (i.e., approximately one third less than previously thought). Further, our work also summarizes the relationship of work sample exams to measures of general cognitive ability. We found that work sample tests were associated with an observed correlation of .32 with tests of general cognitive ability.

Gender Differences in Work Sample Assessments: Not All Tests Are Created Equal

“…That is, rather than being designed to assess a single underlying construct (such as assessments of cognitive ability or personality), they are designed to have high content validity, assessing proficiencies in tasks required for successful performance on the job, and therefore assess a range of constructs (Roth, Bobko, McFarland, & Buster, 2008). Meta-analytic reviews have indicated that work samples have high criterion-related validity for predicting job performance (Roth, Bobko, & McFarland, 2005;Schmidt & Hunter, 1998). Furthermore, as work samples have a higher degree of fidelity to tasks performed on the job than traditional individual differences predictors (Asher & Sciarrino, 1974), they are generally associated with more favorable applicant reactions (e.g., Hausknecht, Day, & Thomas, 2004;Macan, Avedon, Paese, & Smith, 1994).…”

Work Sample Tests: Their Relationship with Job Performance and Job Experience

“…More recently, the magnitude of the validity coefficients (corrected for criterion unreliability but not for range restriction) obtained in a study based on meta-analytic procedures conducted by Roth et al's (2005) was substantially lower (r= .33) than the magnitude of the estimates resulting from the initial work of Hunter and Hunter (1984) (r=.54). Based on these results, Roth et al (2005) emphasized that work sample validity could be noticeably lower than previously thought and therefore some organizations may be overestimating the validity properties of work samples. Despite the lower validity estimates that were obtained in more recent studies, which have overcome important limitations related with conceptual and methodological problems (Asher & Sciarrino, 1974) on the one hand, and the limited scope on the other, in general, the directionality of the results of the studies converge on an indication that work sample tests are valid predictors of job performance.…”

“…Schmidt and Hunter (1998) made a contribution to this topic by reporting a correlation of .38 between general mental ability and work samples. Roth et al (2005) indicated an observed correlation of .32 (K=43, N=17,563) which rose to .38 when the estimation was corrected for work sample unreliability and .40 when both types of tests were corrected for unreliability. Schmidt and Hunter's model suggests that cognitive ability impacts on work sample performance through its effects on the acquisition of job knowledge, because cognitive ability increases the speed of acquiring job knowledge processes (Schmidt, Hunter, & Outerbridge, 1986).…”

“…Internships and probationary periods are examples of this last kind of test and therefore cannot be classified as work samples (Heneman & Judge, 2003;Roth, Bobko, & McFarland, 2005). Finally, it is also important to notice that, despite the frequent use of work sample tests as a predictor measure, they can be also used as a criterion measure as well, to validate important organizational outcomes, like training success (Callinan & Robertson, 2000).…”

“…Prior research has demonstrated that work sample tests constitute a selection method that should be placed in the group of the most valid job performance predictors (Hunter & Hunter, 1984;Roth, Bobko, & McFarland, 2005;Schmidt & Hunter, 1998). The validity of work samples is recognized both by researchers and the professionals involved in selection activities, such as human resources practitioners and managers (Terpstra, Kethley, & Foley, 2000).…”