Factors which Improve the Construct Validity of Assessment Centers: A Review

Lievens, Filip

doi:10.1111/1468-2389.00085

Cited by 91 publications

(114 citation statements)

References 41 publications

Supporting

Mentioning

106

Contrasting

Unclassified

Order By: Relevance

“…However, empirical findings accumulated over the last 20 years do not support this theoretical architecture-correlations between different dimensions within the same exercise tend to be much larger than the correlations between dimension across different exercises (Woehr & Arthur, 1999). In other words, exercise effects dominate over dimension effects on PEDRs and, as has been documented by several reviews (e.g., Lievens, 1998;Lievens & Conway, 2001;Sackett & Tuzinski, 2001;Woehr & Arthur, 1999), various AC design modifications have had little or no effect on this basic pattern of findings. Rather, and contrary to the original design of the AC architecture, candidate performance seems to be relatively undifferentiated across the various dimensions defined for each exercise and cross-situationally specific as exercises define different performance situations.…”

Section: Discussionmentioning

confidence: 94%

Revised Estimates of Dimension and Exercise Variance Components in Assessment Center Postexercise Dimension Ratings.

Lance

Lambert

Gewin

et al. 2004

Journal of Applied Psychology

Self Cite

106

180

View full text Add to dashboard Cite

The authors reanalyzed assessment center (AC) multitrait-multimethod (MTMM) matrices containing correlations among postexercise dimension ratings (PEDRs) reported by F. Lievens and J. M. Conway (2001). Unlike F. Lievens and J. M. Conway, who used a correlated dimension-correlated uniqueness model, we used a different set of confirmatory-factor-analysis-based models (1-dimension-correlated Exercise and 1-dimension-correlated uniqueness models) to estimate dimension and exercise variance components in AC PEDRs. Results of reanalyses suggest that, consistent with previous narrative reviews, exercise variance components dominate over dimension variance components after all. Implications for AC construct validity and possible redirections of research on the validity of ACs are discussed.

show abstract

Section: Discussionmentioning

confidence: 94%

Revised Estimates of Dimension and Exercise Variance Components in Assessment Center Postexercise Dimension Ratings.

Lance

Lambert

Gewin

et al. 2004

Journal of Applied Psychology

Self Cite

106

180

View full text Add to dashboard Cite

show abstract

“…On the basis of prior research, several design considerations have been suggested for increasing the construct-related validity of ACs (see Lievens, 1998;Lievens & Conway, 2001;Woehr & Arthur, 2003). Consider, for instance, the suggestion to limit the number of exercises.…”

Section: Discussionmentioning

confidence: 99%

“…Consider, for instance, the suggestion to limit the number of exercises. Whereas a more diverse set of exercises seems to reduce the convergence of dimension ratings across exercises (i.e., lower convergent validity; Lievens, 1998;Schneider & Schmitt, 1992), the opposite might be true for criterion-related validity (i.e., a more diverse set of job-related exercises might increase criterion-related validity). We need to test these predictions by integrating different validity designs in the future.…”

Section: Discussionmentioning

confidence: 99%

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

2009

Self Cite

View full text Add to dashboard Cite

This study presents a simultaneous examination of multiple evidential bases of the validity of assessment center (AC) ratings. In particular, we combine both construct-related and criterion-related validation strategies in the same sample to determine the relative importance of exercises and dimensions. We examine the underlying structure of ACs in terms of exercise and dimension factors while directly linking these factors to a work-related criterion (salary). Results from an AC (N = 753) showed that exercise factors not only explained more variance in AC ratings than dimension factors but also were more important in predicting salary. Dimension factors explained a smaller albeit significant portion of the variance in AC ratings and had lower validity for predicting salary. The implications of these findings for AC theory, practice, and research are discussed.In assessment centers (ACs), trained assessors observe and evaluate candidates' behaviors in job-related exercises. Although the makeup of these high-fidelity simulations differs considerably across ACs, they can be brought back to five generic types: in-baskets, case analyses, role-plays, oral presentations, and group discussions (Bowler & Woehr, 2006;Lievens, Chasteen, Day, & Christiansen, 2006). Another hallmark of ACs is that candidates are rated on job-related dimensions. Recent taxonomic work has grouped this variety of dimensions into six broad psychological constructs: communication, consideration/awareness of others, drive, influencing others, organizing and planning, and problem solving (Arthur, Day, McNelly, & Edens, 2003).Over the last years, these exercise and dimension taxonomies have served as useful frameworks for directing research on the validity of AC ratings. One stream of research has examined the ability of AC ratings to predict external work-related outcomes such as job performance, promotion, managerial potential, and salary. A meta-analysis of this strand of studies showed that ratings on the AC constructs of organizing and planning, problem solving, and influencing others emerged as the most valid predictors of job performance . Another voluminous stream of studies has focused on the internal structure of AC ratings and investigated whether AC ratings indeed reflect candidates' standing on dimensions as measured by multiple

show abstract

“…As is common in ACs, the configuration here was not fully crossed in that not every dimension was assessed in every exercise (see Appendix, Table A3). In accordance with design suggestions about minimizing cognitive load, the number of dimensions assessed within any given exercise was kept to a minimum (Chan, 1996;Lievens, 1998). Because this was a high-stakes evaluation, dimensions were not revealed to participants prior to the AC taking place.…”

Section: Ac Design and Developmentmentioning

confidence: 99%

“…Our AC was, however, developed in keeping with international guidelines (International Task Force on Assessment Center Guidelines, 2009; International Taskforce on Assessment Center Guidelines, 2015) as well as with guidelines in the literature for the development of dimensions (Arthur et al, 2003;Guenole et al, 2013;Lievens, 1998), exercises (Thornton & Mueller-Hanson, 2004), and on training for ACs (Gorman & Rentsch, 2009;Macan et al, 2011).…”

Section: Limitations and Future Directionsmentioning

confidence: 99%

Everything that you have ever been told about assessment center ratings is confounded.

Jackson

Michaelides

Dewberry

et al. 2016

Journal of Applied Psychology

View full text Add to dashboard Cite

Despite a substantial research literature on the influence of dimensions and exercises in assessment centers (ACs), the relative impact of these two sources of variance continues to raise uncertainties because of confounding. With confounded effects, it is not possible to establish the degree to which any one effect, including those related to exercises and dimensions, influences AC ratings. In the current study (N = 698) we used Bayesian generalizability theory to unconfound all of the possible effects contributing to variance in AC ratings. Our results show that ≤ 1.11% of the variance in AC ratings was directly attributable to behavioral dimensions, suggesting that dimension-related effects have no practical impact on the reliability of ACs.Even when taking aggregation level into consideration, effects related to general performance and exercises accounted for almost all of the reliable variance in AC ratings. The implications of these findings for recent dimension-and exercise-based perspectives on ACs are discussed. When, in the context of selection, appraisal, and development, behavioral criteria are used to evaluate individuals, it is essential that these criteria are measured reliably.Unsurprisingly, therefore, the measurement properties of assessment center (AC) ratings have come under close scrutiny in the applied psychology literature. In ACs, the behavior of jobholders or candidates is sampled across several work-related situations (exercises, e.g., a role play exercise, group discussion, presentation) and is typically assessed by trained assessors in terms of pre-defined behavioral dimensions (e.g., communication skills, teamwork, planning and organizing). As a result of their multifaceted measurement properties, incorporating dimensions, exercises, and assessors, ACs provide a rich source of information about the extent to which work-related behavioral criteria can be reliably measured in a job-relevant setting.Historically, researchers have questioned the extent to which behavioral dimensions are measured reliably in ACs, and have implied that researchers should utilize an exercise-oriented approach to scoring ACs

show abstract

Factors which Improve the Construct Validity of Assessment Centers: A Review

Cited by 91 publications

References 41 publications

Revised Estimates of Dimension and Exercise Variance Components in Assessment Center Postexercise Dimension Ratings.

Revised Estimates of Dimension and Exercise Variance Components in Assessment Center Postexercise Dimension Ratings.

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

Everything that you have ever been told about assessment center ratings is confounded.

Contact Info

Product

Resources

About