Tracing Cognition with Assessment Center Simulations: Using Technology to See in the Dark

Guidry, Brett W.; Rupp, Deborah E.; Lanik, Martin

doi:10.1007/978-1-4614-7681-8_11

Cited by 4 publications

(2 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There exist multiple assessment center practices supported by theory but not empirically tested. For example, theory and practice suggest that motivation, cognitive understanding, and experience are related to assessee performance (e.g., Guidry et al, 2013). However, empirical research has yet to explore these issues in depth.…”

Section: Theory and Practice Converge (But Not Research)mentioning

confidence: 99%

An epistemology for assessment and development: How do we know what we know?

Rupp,

Thornton,

Bisbey

et al. 2024

Ind. Organ. Psychol.

Self Cite

View full text Add to dashboard Cite

To make informed decisions, assessment theorists, researchers, and practitioners can evaluate the overlap among (1) relevant theories, (2) empirical contributions, and (3) best practices. Unfortunately, such a task may seem daunting due to the so-called science-practice gap, which can thwart collaboration among these parties. This paper presents an epistemology for delineating the importance of integrating these three sources of knowledge. We then apply this epistemology to show that our current knowledge of assessment and development topics are well integrated in some places, but still quite lacking in others.

show abstract

Section: Theory and Practice Converge (But Not Research)mentioning

confidence: 99%

An epistemology for assessment and development: How do we know what we know?

Rupp,

Thornton,

Bisbey

et al. 2024

Ind. Organ. Psychol.

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, AC exercises can cost hundreds of thousands of dollars to design and administer, making them prohibitively costly for many organizations (Thornton & Rupp, 2006). Assessors represent a major cost because they must be extensively trained and then take substantial time to observe and rate assessee behavior (Guidry et al, 2013; Wirz et al, 2013).…”

Section: Introductionmentioning

confidence: 99%

Automatic scoring of speeded interpersonal assessment center exercises via machine learning: Initial psychometric evidence and practical guidelines

Hickman

Herde

Lievens

et al. 2023

Int J Selection Assessment

View full text Add to dashboard Cite

Assessment center (AC) exercises such as role-plays have established themselves as valuable approaches for obtaining insights into interpersonal behavior, but they are often considered the "Rolls Royce" of personnel assessment due to their high costs. The observation and rating process comprises a substantial part of these costs. In an exploratory case study, we capitalize on recent advances in natural language processing (NLP) by developing NLP-based machine learning (ML) models to investigate the possibility of automatically scoring AC exercises. First, we compared the convergent-related validity and contamination with word count of ML scores based on models that used different NLP methods to operationalize verbal behavior. Second, for the model that maximized convergence while minimizing contamination with word count (i.e., a model that used both n-grams and Universal Sentence Encoder embeddings as predictors), we investigated the criterion-related validity of its scores. Third, we examined how the interrater reliability of the AC role-play scores affects ML model convergence. To do so, we applied seven NLP methods to 96 assessees' transcriptions and trained 10 sets of ML models across 18 speeded AC role-plays to automatically score assessee performance. Results suggest that ML scores recovered most of the original variance in the overall assessment ratings, and replacing one or more human assessors with ML scores maintained criterion-related validity. Additionally, ML models seemed to exhibit higher convergence when assessors consistently detected and utilized observable behaviors to make ratings (i.e., when interrater reliability was higher). Finally, we provide a step-by-step guide for practitioners seeking to implement ML scoring in ACs.

show abstract

Simulations for Personnel Selection

Fetzer¹,

Tuzinski²

2013

View full text Add to dashboard Cite

Tracing Cognition with Assessment Center Simulations: Using Technology to See in the Dark

Cited by 4 publications

References 50 publications

An epistemology for assessment and development: How do we know what we know?

An epistemology for assessment and development: How do we know what we know?

Automatic scoring of speeded interpersonal assessment center exercises via machine learning: Initial psychometric evidence and practical guidelines

Simulations for Personnel Selection

Contact Info

Product

Resources

About