Constructed response formats and their effects on minority–majority differences and validity.

Abstract

The inflow of immigrants challenges organizations to consider alternative selection procedures that reduce potential minority (immigrants)-majority (natives) differences, while maintaining valid predictions of performance. To deal with this challenge, this paper proposes response format as a practically and theoretically relevant factor for situational judgment tests (SJTs). We examine a range of response format categories (from traditional multiple-choice formats to more innovative constructed response formats) and conceptually link these response formats to mechanisms underlying minority-majority differences. Two field experiments are conducted with SJTs. Study 1 (274 job seekers) contrasts minority-majority differences in scores on a multiple-choice versus a written constructed response format. Written constructed responses produce much smaller minority-majority differences (d ϭ .28 vs. d ϭ .92). In Study 2 (269 incumbents), scores on a written constructed versus an audiovisual constructed format are compared. The audiovisual format further reduces minority-majority differences (d ϭ .09 vs. d ϭ .41), with validities remaining the same. Results are suggestive of cognitive load as a contributor to the reduction in minority-majority differences, as are rater effects: Scores of raters evaluating transcribed audiovisual responses, which anonymized test takers, produce larger differences. In sum, altering response modality via more realistic response formats (i.e., the audiovisual constructed format) leads to significant reductions in minority-majority differences without impairing criterion-related validity. Implications for selection theory and practice are discussed.

Subgroup differences in situational judgment test scores: Evidence from large applicant samples

“…Such a constructed response multimedia test does not seem to correlate significantly with cognitive ability (De Soete, Lievens, Oostrom, & Westerveld, ; Lievens, De Corte, & Westerveld, ; Oostrom, Born, Serlie, & van der Molen, ) and produces small to moderate subgroup differences (Cucina, Su, Busciglio, Harris Thomas, & Thompson Peyton, ; De Soete et al, ; Lievens, Sackett, Dahlke, Oostrom, & De Soete, ; Lievens et al, ; Oostrom, Born, Serlie, & van der Molen, , ). In a recent study, Lievens et al () empirically demonstrated that reduced majority–minority subgroup differences in constructed response multimedia tests could be attributed to unintended cognitive load in tests that aim to assess behavior related to the interpersonal domain. Constructed response multimedia tests might require less cognitive resources to perform well because they allow to respond if at least the core of the situation is understood (Hakel, ; Ryan & Greguras, ).…”

Enhancing Personality Assessment in the Selection Context: A Study Protocol on Alternative Measures and an Extended Bandwidth of Criteria

“…Personality is typically assessed via self-report inventories, which face three major limitations in the selection context: (1) a lack of contextualization, (2) relying on applicants as the only source of information, and (3) a close-ended response format (Connelly and Ones, 2010;Oh et al, 2011;Shaffer and Postlethwaite, 2012;Lievens and Sackett, 2017;Lievens et al, 2019). Contextualization describes the degree to which a measurement method refers to a specific situation or context, such as the work context.…”

Situational judgement tests for selection: Traditional vs construct‐driven approaches

“…Yet, this difference between SJT scores in the two instructions was found only in research contexts and non‐high‐stakes contexts. In a high‐stakes context all candidates seem to adopt a ‘ should do’ response frame, regardless of the actual instruction . The scoring strategy may also influence the impact of faking attempts …”

“…Regarding the widening participation agenda, further research is also required to understand how SJT construction, including scoring method and implementation, may impact on under‐represented groups. It is already recognised that such effects are sensitive to both SJT construction and scoring approaches …”