Scoring method of a Situational Judgment Test: influence on internal consistency reliability, adverse impact and correlation with personality?

Abstract

Situational Judgment Tests (SJTs) are increasingly used for medical school selection. Scoring an SJT is more complicated than scoring a knowledge test, because there are no objectively correct answers. The scoring method of an SJT may influence the construct and concurrent validity and the adverse impact with respect to non-traditional students. Previous research has compared only a small number of scoring methods and has not studied the effect of scoring method on internal consistency reliability. This study compared 28 different scoring methods for a rating SJT on internal consistency reliability, adverse impact and correlation with personality. The scoring methods varied on four aspects: the way of controlling for systematic error, and the type of reference group, distance and central tendency statistic. All scoring methods were applied to a previously validated integrity-based SJT, administered to 931 medical school applicants. Internal consistency reliability varied between .33 and .73, which is likely explained by the dependence of coefficient alpha on the total score variance. All scoring methods led to significantly higher scores for the ethnic majority than for the non-Western minorities, with effect sizes ranging from 0.48 to 0.66. Eighteen scoring methods showed a significant small positive correlation with agreeableness. Four scoring methods showed a significant small positive correlation with conscientiousness. The way of controlling for systematic error was the most influential scoring method aspect. These results suggest that the increased use of SJTs for selection into medical school must be accompanied by a thorough examination of the scoring method to be used.

Selection methods in healthcare professions: where are we now and where next?

“…Research presented in this issue advances the literature across a wide range of topics in this area, including analysis of the political validity (stakeholder acceptance) of the H-PAT Ireland aptitude test (Kelly and O'Flynn 2016); predictive validity of computer-based MMIs (CASPer, Dore et al 2016); participation in extra-curricular activities (UrlingsStrop et al 2016) and scoring methods for SJTs (De Leng et al 2016).…”

“…Considering validity and quality from a different perspective, De Leng et al (2016) provide data which demonstrate how the scoring method applied for an SJT can impact on internal consistency reliability, adverse impact and correlations with personality dimensions. They conclude that the way of controlling for systematic error was the most influential scoring method aspect.…”

Response distortion on personality tests in applicants: comparing high-stakes to low-stakes medical settings

“…Despite their importance, it remains unclear how best to measure these characteristics, particularly in the context of high-stakes testing where applicants may be motivated to distort their responses (Albanese et al, 2003;Bore et al, 2009;Musson, 2009;Patterson et al, 2016). In recent years, a variety of selection tools have been developed that aim to assess these noncognitive characteristics, including situational judgment tests (Bore et al, 2009;De Leng et al, 2017;Lievens, 2013;Patterson et al, 2012;Patterson et al, 2009), multiple mini-interviews (Eva et al, 2014;Eva et al, 2009;Eva et al, 2004;Griffin et al, 2012a;Kulasegaram et al, 2010), emotional intelligence tests (Libbrecht et al, 2014), and personality tests (Griffin et al, 2012b;MacKenzie et al, 2017;Rothstein et al, 2006). The current study focuses on personality testing in the context of the high-stakes selection of medical interns assessing the degree to which response distortion might limit the utility of personality tests.…”

Faking on a situational judgment test in a medical school selection setting: Effect of different scoring methods?

“…Whether differences in extreme responding relate to differences in the SJT score is likely to depend on the method used for scoring the SJT. Of the many SJT scoring methods that exist (De Leng et al, ), most use consensus judgment to determine the scoring key (McDaniel, Psotka, Legree, Yost, & Weekley, ) and calculate the distance on the rating scale between an individual's judgment and the consensus judgment. Prior research has demonstrated that these scoring methods may be affected by response tendencies (e.g., extreme response style), introducing a source of systematic error, which may decrease the criterion‐related validity of an SJT (McDaniel et al, ; Weng, Yang, Lievens, & McDaniel, ).…”