A general response process theory for situational judgment tests.

Abstract

Situational judgment tests (SJTs) have emerged as a staple of assessment methodologies for organizational practitioners and researchers. Despite their prevalence, many questions regarding how to interpret respondent choices or how variations in item construction and instruction influence the nature of observed responses remain. Existing conceptual and empirical efforts to explore these questions have largely been rooted in reflexive psychometric measurement models that describe participant responses as indicative of (usually multiple) latent constructs. However, some have suggested that a key to better understanding SJT responses lies in unpacking the judgment and decision-making processes employed by respondents and the psychological and contextual factors that shape how those processes play out. To this end, the present paper advances an integrative and generalizable process-oriented theory of SJT responding. The framework, labeled Situated Reasoning and Judgment (SiRJ), proposes that respondents engage in a series of conditional reasoning, similarity, and preference accumulation judgments when deciding how to evaluate and respond to an SJT item. To evaluate the theory's plausibility and utility, the SiRJ framework is translated into a formal computational model and results from a simulation study are analyzed using neural network and Bayesian survival analytic techniques that demonstrate its capability to replicate existing and new empirical effects, suggest insights into SJT interpretation and development, and stimulate new directions for future research. An interactive web application that allows users to explore the computational model

Do situational characteristics affect the validity of personality situational judgment items?

“…Though SJT response options clearly play the main role in the types of items we examined, as interactionist models for real‐world behavior suggest (Mischel & Shoda, 1995; Tett & Burnett, 2003; Tett & Guterman, 2000), and parallel research finds for personality Likert‐type measures (Shaffer & Postlethwaite, 2012), situational cues can play an important supporting role. This is also consistent with current theories of the SJT response process (e.g., Grand, 2020; Harris et al, 2016; Martin‐Raugh & Kell, 2021), which propose that stems help orient test takers. Taken together, our study and Reeder's earlier work seem to suggest that including situational cues pertaining to interpersonal demands (both studies) and role clarity (ours) could increase convergent validity for achievement‐striving/conscientiousness items, whereas social pressures (Reeder) and emotional challenges (Reeder) could decrease it.…”

“…However, from a construct validity perspective, it is unclear what construct(s) the items measure, as they do not yield internally consistent and interpretable factors (Christian et al, 2010; Lievens et al, 2008). Researchers have urged a better understanding of SJTs’ construct validity (e.g., Brooks & Highhouse, 2006; Christian et al, 2010; Grand, 2020).…”

“…For personality measurement specifically, contextualizing statements with “at work” or “in school” tags increases validity, as this prompts individuals to consider their behavior in the performance setting of interest (Shaffer & Postlethwaite, 2012). Relatedly, many view SJT stems as integral to the method because they orient test takers to situations to then elicit their related knowledge or behavioral preferences, typically by having test‐takers interact with the corresponding response options (e.g., Campion & Ployhart, 2013; Gessner & Klimoski, 2006; Grand, 2020; Harris et al, 2016).…”

“…Test‐takers may still be able to identify the best answers to stemless items and associated SJT scores may exhibit similar levels of reliability and validity as scores based on intact items (Krumm et al, 2015; Schäpers, Freudenstein, et al, 2020; Schäpers, Mussel, et al, 2020). In response, prostem researchers have argued that stems provide relevant information for making judgments and that when they are removed, test‐takers look for clues in the response options to reconstruct the situational context (Grand, 2020; Harris et al, 2016; Martin‐Raugh & Kell, 2021). Indeed, Schäpers, Lievens, et al (2020) found that for some items, raters could reproduce a lot of the context based on clues in the responses.…”

“…Indeed, Schäpers, Lievens, et al (2020) found that for some items, raters could reproduce a lot of the context based on clues in the responses. Forcing test takers to reconstruct the situational context may have negative implications for their experience: stemless items may be less engaging (Schäpers, Mussel, et al, 2020c), be experienced as more cognitively taxing (Grand, 2020), and result in lower scores (Krumm et al, 2015; Schäpers, Lievens, et al, 2020, Schäpers, Mussel, et al, 2020). An additional argument against stem removal is that the ability to correctly perceive the demands of the situation could provide incremental validity beyond that provided by scores based on endorsing the “right” options (Rockstuhl et al, 2015).…”