Rating Tasks Versus Dimensions in Assessment Centers: A Psychometric Comparison

Jackson, Duncan J. R.; Stillman, Jennifer A.; Atkins, Stephen

doi:10.1207/s15327043hup1803_2

Cited by 36 publications

(55 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Second, we tested an exercises-only model. In this "situationist" model, exercises are the building blocks of ACs that are then conceptualized as a series of miniaturized work samples designed to elicit job-relevant behavior (Jackson, Stillman, & Atkins, 2005;Lowry, 1997;Robertson, Gratton, & Sharpley, 1987;Sackett & Dreher, 1982). Such a model is also consistent with empirical research showing that exercise effects in ACs represent true cross-situational variability of candidates across exercises, and not simply unwanted method variance (Lance, Foster, Gentry, & Thoresen, 2004;Lance et al, 2000).…”

Section: Underlying Structure Of Assessment Center Ratingsmentioning

confidence: 83%

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

2009

View full text Add to dashboard Cite

This study presents a simultaneous examination of multiple evidential bases of the validity of assessment center (AC) ratings. In particular, we combine both construct-related and criterion-related validation strategies in the same sample to determine the relative importance of exercises and dimensions. We examine the underlying structure of ACs in terms of exercise and dimension factors while directly linking these factors to a work-related criterion (salary). Results from an AC (N = 753) showed that exercise factors not only explained more variance in AC ratings than dimension factors but also were more important in predicting salary. Dimension factors explained a smaller albeit significant portion of the variance in AC ratings and had lower validity for predicting salary. The implications of these findings for AC theory, practice, and research are discussed.In assessment centers (ACs), trained assessors observe and evaluate candidates' behaviors in job-related exercises. Although the makeup of these high-fidelity simulations differs considerably across ACs, they can be brought back to five generic types: in-baskets, case analyses, role-plays, oral presentations, and group discussions (Bowler & Woehr, 2006;Lievens, Chasteen, Day, & Christiansen, 2006). Another hallmark of ACs is that candidates are rated on job-related dimensions. Recent taxonomic work has grouped this variety of dimensions into six broad psychological constructs: communication, consideration/awareness of others, drive, influencing others, organizing and planning, and problem solving (Arthur, Day, McNelly, & Edens, 2003).Over the last years, these exercise and dimension taxonomies have served as useful frameworks for directing research on the validity of AC ratings. One stream of research has examined the ability of AC ratings to predict external work-related outcomes such as job performance, promotion, managerial potential, and salary. A meta-analysis of this strand of studies showed that ratings on the AC constructs of organizing and planning, problem solving, and influencing others emerged as the most valid predictors of job performance . Another voluminous stream of studies has focused on the internal structure of AC ratings and investigated whether AC ratings indeed reflect candidates' standing on dimensions as measured by multiple

show abstract

Section: Underlying Structure Of Assessment Center Ratingsmentioning

confidence: 83%

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

2009

View full text Add to dashboard Cite

show abstract

“…Based on these findings, some have advocated the abandonment of dimensions in favor of task‐based rating strategies (Jackson, Stillman, & Atkins, 2005; Lance, 2008a; Lance, Foster, Gentry, & Thoresen, 2004; Sackett & Dreher, 1982). From this perspective, the exercise structure characterizing PEDRs is not attributable to measurement bias but instead represents valid situational specificity in performance (Lance, Foster et al, 2004).…”

mentioning

confidence: 99%

Exercises and Dimensions Are the Currency of Assessment Centers

Hoffman

Melchers

Blair³

et al. 2011

Personnel Psychology

View full text Add to dashboard Cite

A novel assessment center (AC) structure that models broad dimension factors, exercise factors, and a general performance factor is proposed and supported in 4 independent samples of AC ratings. Consistent with prior research, the variance attributable to dimension and exercise factors varied widely across ACs. To investigate the construct validity of these empirically supported components of AC ratings, the nomological network of broad dimensions, exercises, and general performance was examined. Results supported the criterion‐related validity of broad dimensions and exercises as predictors of effectiveness and success criteria as well as the incremental validity of broad dimensions beyond exercises and general performance. Finally, the relationships between individual differences and AC factors supported the construct validity of broad dimension factors and provide initial insight as to the meaning of exercise specific variance and general AC performance.

show abstract

“…Specifically, the dimension main effect accounted for roughly 21% of the total variance, the person effect accounted for 18%, and the person by dimension effect accounted for 20% of the total variance. In contrast, the exercise effect accounted for less than 1% of the variance, and the person by exercise effect accounted for roughly 5% percent of the total variance.Similarly, Jackson et al (2005) examined the relative variance accounted for by person, dimension, and exercise factors (including the associated two-way interactions). Their results indicated that that the person effect, dimension effect, and person by dimension interaction accounted for approximately 36% of the variance (31.9%, 2.2%, and 1.8%, respectively).…”

Section: Previous Empirical Applicationsmentioning

confidence: 99%

“…Despite the plethora of studies conducted on the assessment center construct validity paradox, few have chosen to utilize generalizability theory to examine what role exercises and dimensions play in assessment center PEDRs (Arthur et al, 2000;Jackson, Stillman, & Atkins, 2005). The present study seeks to rectify this shortcoming by applying a hybrid of Monte Carlo data generation and generalizability theoryin an effort to summarize the current state of assessment center without utilizing a problematic methodology.…”

Section: Reevaluating Assessment Center Construct-related Validitymentioning

confidence: 99%

A Reevaluation of Assessment Center Construct-Related Validity

Cahoon¹,

Bowler²,

Bowler³

2012

IJBM

View full text Add to dashboard Cite

Recent Monte Carlo research (Lance, Woehr, & Meade, 2007) has questioned the primary analytical tool used to assess the construct-related validity of assessment center post-exercise dimension ratings (PEDRs) -a confirmatory factor analysis of a multitrait-multimethod (MTMM) matrix. By utilizing a hybrid of Monte Carlo data generation and univariate generalizability theory, we examined three primary sources of variance (i.e., persons, dimensions, and exercises) and their interactions in 23 previously published assessment center MTMM matrices. Overall, the person, dimension, and person by dimension effects accounted for a combined 34.06% of variance in assessment center PEDRs (16.83%, 4.02%, and 13.21%, respectively). However, the largest single effect came from the person by exercise interaction (21.83%). Implications and suggestions for future assessment center research and design are discussed.

show abstract

Rating Tasks Versus Dimensions in Assessment Centers: A Psychometric Comparison

Cited by 36 publications

References 49 publications

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

Exercises and Dimensions Are the Currency of Assessment Centers

A Reevaluation of Assessment Center Construct-Related Validity

Contact Info

Product

Resources

About