The importance of situation evaluation and the ability to identify criteria in a construct-driven situational judgment test

Wang, Dan; Oostrom, Janneke K.; Schollaert, Eveline

doi:10.1016/j.paid.2023.112182

Cited by 5 publications

(13 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, several design choices may have influenced results of Study 1 and call for further research. First, we examined the relationship between ATIC and SJT performance on the item level (in contrast to Melchers & Hupp [2017], Wang et al [2023]). Administering a variety of SJT items across several constructs was beneficial for the generalizability of our results across items and constructs.…”

Section: Discussionmentioning

confidence: 99%

“…However, ATIC adopts a different reference point. While the situational judgment in SJTs describes how an individual perceives and categorizes a given situation, ATIC describes whether an individual correctly recognizes on which criteria they are evaluated in different situations (see also Wang et al, 2023).…”

Section: Theoretical Backgroundmentioning

confidence: 99%

“…Conversely, Wolcott et al (2021) administered an empathy SJT to a small sample and found no relationship between ATIC and SJT performance. The most comprehensive research on the relevance of ATIC in SJTs was presented by Wang et al (2023).…”

Section: Theoretical Backgroundmentioning

confidence: 99%

“…Second, we employ a broad set of items from several SJTs. Third, we use the traditional and most common way of assessing ATIC (as opposed to the multiple-choice ATIC measure used by Wang et al [2023]). Moreover, we analyze data on the SJT item level (Study 1), thereby accounting for the multidimensionality that is typically evident between and within SJTs (Tiffin et al, 2020;Whetzel et al, 2020) as well as on the test level (Study 2), thereby following the more typical approach of other studies on ATIC (Ingold et al, 2015;Klehe et al, 2012;Kleinmann et al, 2011).…”

Section: Theoretical Backgroundmentioning

confidence: 99%

“…Indeed, the ability to identify criteria (ATIC; Kleinmann, 1993), defined as an individual's capability to see through nontransparent selection procedures and identify the psychological construct that is being assessed (e.g., Kleinmann et al, 2011), was found to be of substantial relevance in assessment centers and situational interviews (Ingold et al, 2015;Jansen et al, 2013;König et al, 2007). However, while the relevance of ATIC is fairly well established for the aforementioned simulations, much less is known about the relevance of ATIC in SJTs (for an exception, see Wang et al [2023]). In the current paper, we build on prior theorizing as well as on extant empirical evidence (for details, see below) and examine whether ATIC predicts SJT performance across two studies with two different operationalizations of ATIC in SJTs.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Does understanding what a test measures make a difference? On the relevance of the ability to identify criteria for situational judgment test performance

Reznik,

Krumm,

Freudenstein

et al. 2023

Int J Selection Assessment

View full text Add to dashboard Cite

Situational judgment tests (SJTs) are low‐fidelity simulations that are often used in personnel selection. Previous research has provided evidence that the ability to identify criteria (ATIC)—individuals' capability to detect underlying constructs in nontransparent personnel selection procedures—is relevant in simulations in personnel selection, such as assessment centers and situational interviews. Building on recent theorizing about response processes in SJTs as well as on previous empirical results, we posit that ATIC predicts SJT performance. We tested this hypothesis across two preregistered studies. In Study 1, a between‐subjects planned‐missingness design (N = 391 panelists) was employed and 55 selected items from five different SJTs were administered. Mixed‐effects‐modeling revealed a small effect for ATIC in predicting SJT responses. Results were replicated in Study 2 (N = 491 panelists), in which a complete teamwork SJT was administered with a high‐ or a low‐stakes instruction and showed either no or a small correlation with ATIC, respectively. We compare these findings with other studies, discuss implications for our understanding of response processes in SJTs, and derive avenues for future research.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Theoretical Backgroundmentioning

confidence: 99%

Section: Theoretical Backgroundmentioning

confidence: 99%

Section: Theoretical Backgroundmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Does understanding what a test measures make a difference? On the relevance of the ability to identify criteria for situational judgment test performance

Reznik,

Krumm,

Freudenstein

et al. 2023

Int J Selection Assessment

View full text Add to dashboard Cite

show abstract

Validity and reliability of the situational judgment test instrument for assessing the ‘Pancasila’ student profile

Wahyuni,

Rahayu,

Naga

2024

Cogent Education

View full text Add to dashboard Cite

Situational judgment test measures administrator computational thinking with factor analysis

Indrawati,

Permansah,

Ninghardjanti

et al. 2024

IJECE

View full text Add to dashboard Cite

Computational thinking skills (CTS) play a crucial role across diverse domains, involving a thinking process that allows the execution of solutions by information processing agents. Measuring the level of CTS becomes essential to ensure that administrators effectively leverage technology. However, finding suitable instruments to measure and justify CTS levels in administration can be challenging. The selection of situational judgement test (SJT) is supported by its validity and reliability in assessing attributes, including professionalism. The instrument’s development includes face validity, discriminant validity (using Pearson correlation and Cronbach’s alpha), and exploratory factor analysis (EFA). The study involved 111 undergraduate administration students from various Indonesian universities, and data were collected in 2023. Following a discriminant validity analysis, ten items were eliminated, while 23 met the criteria with p0.185. Subsequently, EFA yielded 16 items forming seven components, covering algorithmic thinking, problem-solving, technology literacy, problem abstraction, problem reformulation, data management in administration technology, and administrative data presentation with loading factor variations (0.421-0.868). The final instrument, consisting of 16 valid items and seven components, effectively evaluates the level of administrator computational thinking skills (ACTS) among administration students.

show abstract

The importance of situation evaluation and the ability to identify criteria in a construct-driven situational judgment test

Cited by 5 publications

References 47 publications

Does understanding what a test measures make a difference? On the relevance of the ability to identify criteria for situational judgment test performance

Does understanding what a test measures make a difference? On the relevance of the ability to identify criteria for situational judgment test performance

Validity and reliability of the situational judgment test instrument for assessing the ‘Pancasila’ student profile

Situational judgment test measures administrator computational thinking with factor analysis

Contact Info

Product

Resources

About