The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

Abstract

This study presents a simultaneous examination of multiple evidential bases of the validity of assessment center (AC) ratings. In particular, we combine both construct-related and criterion-related validation strategies in the same sample to determine the relative importance of exercises and dimensions. We examine the underlying structure of ACs in terms of exercise and dimension factors while directly linking these factors to a work-related criterion (salary). Results from an AC (N = 753) showed that exercise factors not only explained more variance in AC ratings than dimension factors but also were more important in predicting salary. Dimension factors explained a smaller albeit significant portion of the variance in AC ratings and had lower validity for predicting salary. The implications of these findings for AC theory, practice, and research are discussed.In assessment centers (ACs), trained assessors observe and evaluate candidates' behaviors in job-related exercises. Although the makeup of these high-fidelity simulations differs considerably across ACs, they can be brought back to five generic types: in-baskets, case analyses, role-plays, oral presentations, and group discussions (Bowler & Woehr, 2006;Lievens, Chasteen, Day, & Christiansen, 2006). Another hallmark of ACs is that candidates are rated on job-related dimensions. Recent taxonomic work has grouped this variety of dimensions into six broad psychological constructs: communication, consideration/awareness of others, drive, influencing others, organizing and planning, and problem solving (Arthur, Day, McNelly, & Edens, 2003).Over the last years, these exercise and dimension taxonomies have served as useful frameworks for directing research on the validity of AC ratings. One stream of research has examined the ability of AC ratings to predict external work-related outcomes such as job performance, promotion, managerial potential, and salary. A meta-analysis of this strand of studies showed that ratings on the AC constructs of organizing and planning, problem solving, and influencing others emerged as the most valid predictors of job performance . Another voluminous stream of studies has focused on the internal structure of AC ratings and investigated whether AC ratings indeed reflect candidates' standing on dimensions as measured by multiple

Assessment Center Practices in South Africa

“…In the recent meta-analysis by Dilchert and Ones (2009) the relevance of these dimensions for job performance was supported: The best dimensional predictor for job performance was problem solving, followed by influencing others, and organizing and planning, and communication skills. Given that, it should not be too surprising that these four dimensions were also predictive for a work-related criterion (salary) in a recent study by Lievens, Dilchert, and Ones (2009).…”

Interpersonal Dynamics in Assessment Center Exercises

“…In addition, there is now relative consensus that this substantial exercise variance does not represent measurement bias but true cross-situational performance differences of participants across exercises (Lance, 2008;Lance, Hoffman, Gentry, & Baranik, 2008;Lievens, 2002;Lievens, Dilchert, & Ones, 2009). This is because AC exercises present different situational demands to participants, thereby producing variability in performance across exercises (Gibbons & Rupp, 2009;Howard, 2008;Putka & Hoffman, 2013).…”

“…The limited research into AC exercises is surprising because a vast body of research has revealed that the largest portions of variance in dimension ratings across exercises in ACs can be attributed to participant performance differences across exercises (also referred to as exercise effects; Kuncel & Sackett, 2014; Lance, Lambert, Gewin, Lievens, & Conway, 2004; Putka & Hoffman, 2013), even though some recent studies also found sizable portions of dimension variance (Hoffman, Melchers, Blair, Kleinmann, & Ladd, 2011; Monahan, Hoffman, Lance, Jackson, & Foster, 2013). In addition, there is now relative consensus that this substantial exercise variance does not represent measurement bias but true cross-situational performance differences of participants across exercises (Lance, 2008; Lance, Hoffman, Gentry, & Baranik, 2008; Lievens, 2002; Lievens, Dilchert, & Ones, 2009). This is because AC exercises present different situational demands to participants, thereby producing variability in performance across exercises (Gibbons & Rupp, 2009; Howard, 2008; Putka & Hoffman, 2013).…”

Investigating the construct validity of an electronic in-basket exercise using bias-corrected bootstrapping and Monte Carlo re-sampling techniques

“…According to Arthur, Day and Woehr (2008), validity must be established at the start of test construction and prior to operational use. Lievens, Dilchert and Ones (2009) defined validity as the process of collecting evidence (AC results) to determine the meaning of such assessment ratings and the inferences based on these ratings. Validity is defined by the use and purpose of the AC and is crucial to the permissibility of inferences derived from AC measures.…”

“…The popularity of ACs is because of their many strengths, including that they have little adverse impact and predict a variety of performance criteria (Thornton & Rupp, 2006) with predictive validity correlations ranging from 0.28 to 0.52 (Gaugler, Rosenthal, Thornton, & Bentson, 1987;Hermelin et al, 2007). In addition, the method has been shown to have high criterion-related validity, as well as content validity (Gaugler et al, 1987;Lievens et al, 2009). However, research evidence concerning the internal structure of ACs shows much less support for the construct validity of AC dimensions ( Kleinmann et al, 2011;Tett & Burnett, 2003).…”