Toward an understanding of situational judgment item validity and group differences.

Abstract

This paper evaluates 2 adjustments to common scoring approaches for situational judgment tests (SJTs). These adjustments can result in substantial improvements to item validity, reductions in mean racial differences, and resistance to coaching designed to improve scores. The first adjustment, applicable to SJTs that use Likert scales, controls for elevation and scatter (Cronbach & Gleser, 1953). This adjustment improves item validity. Also, because there is a White-Black mean difference in the preference for extreme responses on Likert scales (Bachman & O'Malley, 1984), these adjustments substantially reduce White-Black mean score differences. Furthermore, this adjustment often eliminates the score elevation associated with the coaching strategy of avoiding extreme responses (Cullen, Sackett, & Lievens, 2006). Item validity is shown to have a U-shaped relationship with item means. This holds both for SJTs with Likert score response formats and for SJTs where respondents identify the best and worst response option. Given the U-shaped relationship, the second adjustment is to drop items with midrange item means. This permits the SJT to be shortened, sometimes dramatically, without necessarily harming validity.

Scoring method of a Situational Judgment Test: influence on internal consistency reliability, adverse impact and correlation with personality?

“…Systematic error in an SJT score may be caused by response tendencies or coaching in strategies on how to use the Likert scale, for example only opt for the extremes or only opt for the middle of the scale (McDaniel et al 2011). Moreover, response tendencies are influenced by ethnic differences.…”

“…Some SJT studies have used the squared distance (McDaniel et al 2011), whereas others have used the absolute distance (Legree 1995). Squaring the distance gives more weight to ratings that deviate more from the reference group (Legree et al 2005).…”

“…Raw consensus does not control for systematic error. McDaniel et al (2011) examined the effect of these three scoring methods on the concurrent validity in two studies, using scores on a biodata scale measuring quitting tendencies and supervisory ratings of job performance as criterion. Higher concurrent validity was found for the standardized consensus and dichotomous consensus scales than for the raw consensus scale, which they explained by the removal of systematic error from the SJT score.…”

“…SJTs with a rational scoring key and a Likert scale response format can be scored using raw, standardized, and dichotomous consensus (McDaniel et al 2011). Raw consensus computes the distance between the applicant's rating and the mean rating of the reference group using the raw data.…”

“…By standardizing or dichotomizing the data, McDaniel et al (2011) attempted to control for systematic error. Systematic error in an SJT score may be caused by response tendencies or coaching in strategies on how to use the Likert scale, for example only opt for the extremes or only opt for the middle of the scale (McDaniel et al 2011).…”