A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings.

Abstract

Two recent reviews have attempted to summarize findings quantitatively regarding assessment center (AC) construct-related validity (i.e., Lance, Lambert, Gewin, Lievens, & Conway, 2004; Lievens & Conway, 2001). Unlike these previous studies, which reanalyzed individual multitrait-multimethod (MTMM) matrices from previously published research, the authors recoded and combined past matrices into a single MTMM matrix. This matrix, comprised of 6 dimensions each measured by 6 exercises, was then analyzed, providing a more generalizable set of results. Both dimensions and exercises were found to contribute substantially to AC ratings. Specific dimensions (i.e., communication, influencing others, organizing and planning, and problem solving) appeared more construct valid than others (i.e., consideration/awareness of others and drive). Implications for AC design and practice are discussed.

Everything that you have ever been told about assessment center ratings is confounded.

“…An issue that has been raised in the AC literature is the possibility that there are often conceptual similarities among the different dimensions used in operational ACs (e.g., as implied in Bowler & Woehr, 2006) and this is likely to be true in the present study. If conceptual similarities between different dimensions are apparent, then this has the potential to inflate person main effects.…”

A Reevaluation of Assessment Center Construct-Related Validity

“…Whereas results from the Lance, Lambert et al (2004) analysis sided with other individual studies that concluded that assessment center PEDRs are a function of exercises and not dimensions (cf. Bycio et al, 1987;Lance et al, 2000;Lance, Foster, et al, 2004;Lievens& Conway, 2001), Bowler and Woehr (2006) and Lievens and Conway (2001) produced results to contrary. Interestingly, the findings of all of these studies were based on the same statistical technique -a CFA of a MTMM matrix.…”

“…Thus, they concluded that exercise effects preside over dimension effects in assessment center PEDRs. Bowler and Woehr (2006) conducted a similar study that also reexamined a similar set of previously reported assessment center MTMM matrices. However, rather than individually reanalyze each MTMM matrix, they chose to recode the data from 35 MTMM matrices into a single MTMM matrix comprised of six dimensions and six exercises.…”

“…Search terms included assessment center in conjunction with multitrait-multimethod and construct validity. In addition to searching the relevant databases, studies that were included in the previous reviews (i.e., Bowler &Woehr, 2006;Lance, Lambert, et al, 2004;Lievens& Conway, 2001) were examined for potential sources of data.…”

“…Arthur et al, 2000;Bycio, Alvares, & Hahn, 1987;Lance, Foster et al, 2004;Lance et al, 2000). In an attempt to address these inconsistencies, three extensive meta-analytic reevaluations of assessment center construct-related validity have been conducted (Bowler &Woehr, 2006;Lance, Lambert et al, 2004;Lievens& Conway, 2001;). In one way or another, all three of these reviews utilized previously published assessment center MTMM matrices.…”