Prediction of Cross-Situationally Consistent and Specific Aspects of Assessment Center Performance

Lance, Charles E.; Foster, Mark R.; Nemeth, Yvette M.; Gentry, William A.; Drollinger, Sabrina Marie

doi:10.1037/e518572013-164

Cited by 14 publications

(27 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, the situational specificity interpretation views exercise effects as representing cross‐situationally specific performance true scores and therefore predicts that exercise factors should be related to external performance‐related variables. The findings of Lance and colleagues (Lance et al, 2000; Lance, Foster, et al, 2004; Lance et al, 2007) from six different ACs strongly supported the situational specificity interpretation of exercise effects. Counter to the method bias predictions but consistent with the situational specificity predictions, exercise factors were much more often than not significantly related to other performance‐related variables that were external to the “core” CFAs.…”

Section: Reconciling the Construct Validity Paradoxsupporting

confidence: 52%

Why Assessment Centers Do Not Work the Way They Are Supposed To

Lance

2008

Ind. organ. psychol.

Self Cite

113

204

View full text Add to dashboard Cite

Assessment centers (ACs) are often designed with the intent of measuring a number of dimensions as they are assessed in various exercises, but after 25 years of research, it is now clear that AC ratings that are completed at the end of each exercise (commonly known as postexercise dimension ratings) substantially reflect the effects of the exercises in which they were completed and not the dimensions they were designed to reflect. This is the crux of the long-standing ''construct validity problem'' for AC ratings. I review the existing research on AC construct validity and conclude that (a) contrary to previous notions, AC candidate behavior is inherently cross-situationally (i.e., cross-exercise) specific, not cross-situationally consistent as was once thought, (b) assessors rather accurately assess candidate behavior, and (c) these facts should be recognized in the redesign of ACs toward task-or rolebased ACs and away from traditional dimension-based ACs.

show abstract

Section: Reconciling the Construct Validity Paradoxsupporting

confidence: 52%

Why Assessment Centers Do Not Work the Way They Are Supposed To

Lance

2008

Ind. organ. psychol.

Self Cite

113

204

View full text Add to dashboard Cite

show abstract

“…Candidate main effects, with respect to SJTs, are analogous to a general judgement factor, but should not be confused with general mental ability/g (e.g., Gonzalez-Mul e, Mount, & Oh, 2014) or with a dominant general factor as generated through principal components analysis (PCA; see Jackson et al, 2015;Lance & Jackson, 2015). Also, candidate main effects in the context of SJTs are different from candidate main effects identified in the AC literature (Lance, Foster, Nemeth, Gentry, & Drollinger, 2007;Putka & Hoffman, 2013). This is because, in SJTs, candidate main effects are concerned with judgements relating to hypothetical situations.…”

Section: Sjt-specific Candidate Main Effectsmentioning

confidence: 98%

The internal structure of situational judgement tests reflects candidate main effects: Not dimensions or situations

Jackson

LoPilato

Hughes

et al. 2016

J Occupat & Organ Psyc

View full text Add to dashboard Cite

Despite their popularity and capacity to predict performance, there is no clear consensus on the internal measurement characteristics of situational judgement tests (SJTs). Contemporary propositions in the literature focus on treating SJTs as methods, as measures of dimensions, or as measures of situational responses. However, empirical evidence relating to the internal structure of SJT scores is lacking. Using generalizability theory, we decomposed multiple sources of variance for three different SJTs used with different samples of job candidates (N1 = 2,320; N2 = 989; N3 = 7,934). Results consistently indicated that (1) the vast majority of reliable observed score variance reflected SJT‐specific candidate main effects, analogous to a general judgement factor, and that (2) the contribution of dimensions and situations to reliable SJT variance was, in relative terms, negligible. These findings do not align neatly with any of the proposals in the contemporary literature; however, they do suggest an internal structure for SJTs. Practitioner points To help optimize reliable variance, overall‐level aggregation should be used when scoring SJTs. The majority of reliable variance in SJTs reflects a general performance factor, relative to variance pertaining to specific dimensions or situations. SJT‐based developmental feedback should be delivered in terms of general SJT performance rather than on performance relating to specific dimensions or situations. Generalizability theory, although underutilized in organizational multifaceted measurement, offers an approach to informing on the psychometric properties of SJTs that is well suited to the complexities of SJT measurement designs.

show abstract

“…In this study, we combined an internal construct-related validity design with a criterion-related validity design. As suggested by several scholars (Arthur et al, 2008;Lance, 2008;Lance, Foster, Nemeth, Gentry, & Drollinger, 2007), future studies might also combine internal and external construct-related validity designs. This means that the internal structure of AC ratings (latent dimension and exercise factors) is linked to constructs measured with other predictor measures (personality inventories, cognitive ability tests, etc.)…”

Section: Discussionmentioning

confidence: 99%

“…In particular, Lance and colleagues (Lance et al, 2000;Lance et al, 2007) have reported correlations between latent exercise factors and a general latent performance factor on one hand and external criteria such as cognitive ability, personality, job knowledge, and job performance on the other. Results showed that exercise factors were differentially related to these external criteria (job knowledge, cognitive ability, and job performance but not personality), whereas the general performance factor was consistently related to all of these criteria.…”

Section: Construct-related and Criterion-related Validitymentioning

confidence: 99%

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

2009

View full text Add to dashboard Cite

This study presents a simultaneous examination of multiple evidential bases of the validity of assessment center (AC) ratings. In particular, we combine both construct-related and criterion-related validation strategies in the same sample to determine the relative importance of exercises and dimensions. We examine the underlying structure of ACs in terms of exercise and dimension factors while directly linking these factors to a work-related criterion (salary). Results from an AC (N = 753) showed that exercise factors not only explained more variance in AC ratings than dimension factors but also were more important in predicting salary. Dimension factors explained a smaller albeit significant portion of the variance in AC ratings and had lower validity for predicting salary. The implications of these findings for AC theory, practice, and research are discussed.In assessment centers (ACs), trained assessors observe and evaluate candidates' behaviors in job-related exercises. Although the makeup of these high-fidelity simulations differs considerably across ACs, they can be brought back to five generic types: in-baskets, case analyses, role-plays, oral presentations, and group discussions (Bowler & Woehr, 2006;Lievens, Chasteen, Day, & Christiansen, 2006). Another hallmark of ACs is that candidates are rated on job-related dimensions. Recent taxonomic work has grouped this variety of dimensions into six broad psychological constructs: communication, consideration/awareness of others, drive, influencing others, organizing and planning, and problem solving (Arthur, Day, McNelly, & Edens, 2003).Over the last years, these exercise and dimension taxonomies have served as useful frameworks for directing research on the validity of AC ratings. One stream of research has examined the ability of AC ratings to predict external work-related outcomes such as job performance, promotion, managerial potential, and salary. A meta-analysis of this strand of studies showed that ratings on the AC constructs of organizing and planning, problem solving, and influencing others emerged as the most valid predictors of job performance . Another voluminous stream of studies has focused on the internal structure of AC ratings and investigated whether AC ratings indeed reflect candidates' standing on dimensions as measured by multiple

show abstract

Prediction of Cross-Situationally Consistent and Specific Aspects of Assessment Center Performance

Cited by 14 publications

References 0 publications

Why Assessment Centers Do Not Work the Way They Are Supposed To

Why Assessment Centers Do Not Work the Way They Are Supposed To

The internal structure of situational judgement tests reflects candidate main effects: Not dimensions or situations

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

Contact Info

Product

Resources

About