Impact of Common Rater Variance on Construct Validity of Assessment Center Dimension Judgments

Kolk, Nanja J.; Born, Marise Ph.; Flier, H. van der

doi:10.1207/s15327043hup1504_02

Cited by 23 publications

(25 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…But the results of Kolk et al (2002) did show a reduction in DDSE correlations for the experimental rating method as compared to the more traditional within‐exercise method. Does that mean that the latter method engenders within‐exercise halo error?…”

mentioning

confidence: 82%

“…Thus, OPRs as one index of assessors’ overall impressions of AC candidates’ performance are clearly performance related and are not merely indicative of performance‐irrelevant halo error . As such, I suggest that although the mean difference in DDSE correlations between the experimental and the control (within‐exercise rating method) groups in the study of Kolk et al (2002) may reflect a reduction in halo error in the experimental group, it may also reflect attenuation in these correlations in the experimental group due to reduced reliability as a function of using a single assessor’s ratings versus some aggregate of multiple assessors’ ratings. Additional research is needed to settle this issue.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Why Assessment Centers Do Not Work the Way They Are Supposed To

Lance

2008

Ind. organ. psychol.

113

204

View full text Add to dashboard Cite

Assessment centers (ACs) are often designed with the intent of measuring a number of dimensions as they are assessed in various exercises, but after 25 years of research, it is now clear that AC ratings that are completed at the end of each exercise (commonly known as postexercise dimension ratings) substantially reflect the effects of the exercises in which they were completed and not the dimensions they were designed to reflect. This is the crux of the long-standing ''construct validity problem'' for AC ratings. I review the existing research on AC construct validity and conclude that (a) contrary to previous notions, AC candidate behavior is inherently cross-situationally (i.e., cross-exercise) specific, not cross-situationally consistent as was once thought, (b) assessors rather accurately assess candidate behavior, and (c) these facts should be recognized in the redesign of ACs toward task-or rolebased ACs and away from traditional dimension-based ACs.

show abstract

mentioning

confidence: 82%

mentioning

confidence: 99%

Why Assessment Centers Do Not Work the Way They Are Supposed To

Lance

2008

Ind. organ. psychol.

113

204

View full text Add to dashboard Cite

show abstract

“…In fact, articles on the internal construct-related validity of ACs typically end by stating that determining the amount of exercise/dimension variance in AC ratings reflects only one side of the equation (Haaland & Christiansen, 2002;Kolk, Born, & van der Flier, 2002;Lievens & Conway, 2001;Robie, Osburn, Morris, Etchegaray, & Adams, 2000;Woehr & Arthur, 2003). Many of these researchers have called for an investigation of how much variance exercises and dimensions explain in an external criterion such as job performance or extrinsic career success (i.e., salary, career progression).…”

Section: Construct-related and Criterion-related Validitymentioning

confidence: 99%

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

2009

View full text Add to dashboard Cite

This study presents a simultaneous examination of multiple evidential bases of the validity of assessment center (AC) ratings. In particular, we combine both construct-related and criterion-related validation strategies in the same sample to determine the relative importance of exercises and dimensions. We examine the underlying structure of ACs in terms of exercise and dimension factors while directly linking these factors to a work-related criterion (salary). Results from an AC (N = 753) showed that exercise factors not only explained more variance in AC ratings than dimension factors but also were more important in predicting salary. Dimension factors explained a smaller albeit significant portion of the variance in AC ratings and had lower validity for predicting salary. The implications of these findings for AC theory, practice, and research are discussed.In assessment centers (ACs), trained assessors observe and evaluate candidates' behaviors in job-related exercises. Although the makeup of these high-fidelity simulations differs considerably across ACs, they can be brought back to five generic types: in-baskets, case analyses, role-plays, oral presentations, and group discussions (Bowler & Woehr, 2006;Lievens, Chasteen, Day, & Christiansen, 2006). Another hallmark of ACs is that candidates are rated on job-related dimensions. Recent taxonomic work has grouped this variety of dimensions into six broad psychological constructs: communication, consideration/awareness of others, drive, influencing others, organizing and planning, and problem solving (Arthur, Day, McNelly, & Edens, 2003).Over the last years, these exercise and dimension taxonomies have served as useful frameworks for directing research on the validity of AC ratings. One stream of research has examined the ability of AC ratings to predict external work-related outcomes such as job performance, promotion, managerial potential, and salary. A meta-analysis of this strand of studies showed that ratings on the AC constructs of organizing and planning, problem solving, and influencing others emerged as the most valid predictors of job performance . Another voluminous stream of studies has focused on the internal structure of AC ratings and investigated whether AC ratings indeed reflect candidates' standing on dimensions as measured by multiple

show abstract

“…Finally, it should be noted that not all assessment center effects were included in our analyses. Previous studies have suggested that individual raters may influence assessment center PEDRs (e.g., Arthur et al, 2000;Kolk, Born, & Flier, 2002). Since multiple ratings are collapsed in the formation of MTMM matrices, the assessment of such an effect was not possible.…”

Section: Limitationsmentioning

confidence: 99%

A Reevaluation of Assessment Center Construct-Related Validity

Cahoon¹,

Bowler²,

Bowler³

2012

IJBM

View full text Add to dashboard Cite

Recent Monte Carlo research (Lance, Woehr, & Meade, 2007) has questioned the primary analytical tool used to assess the construct-related validity of assessment center post-exercise dimension ratings (PEDRs) -a confirmatory factor analysis of a multitrait-multimethod (MTMM) matrix. By utilizing a hybrid of Monte Carlo data generation and univariate generalizability theory, we examined three primary sources of variance (i.e., persons, dimensions, and exercises) and their interactions in 23 previously published assessment center MTMM matrices. Overall, the person, dimension, and person by dimension effects accounted for a combined 34.06% of variance in assessment center PEDRs (16.83%, 4.02%, and 13.21%, respectively). However, the largest single effect came from the person by exercise interaction (21.83%). Implications and suggestions for future assessment center research and design are discussed.

show abstract

Impact of Common Rater Variance on Construct Validity of Assessment Center Dimension Judgments

Cited by 23 publications

References 22 publications

Why Assessment Centers Do Not Work the Way They Are Supposed To

Why Assessment Centers Do Not Work the Way They Are Supposed To

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

A Reevaluation of Assessment Center Construct-Related Validity

Contact Info

Product

Resources

About