Using video- and text-based situational judgement tests for teacher selection: a quasi-experiment exploring the relations between test format, subgroup differences, and applicant reactions

Bardach, Lisa; Rushby, Jade V.; Kim, Lisa; Klassen, Robert M.

doi:10.1080/1359432x.2020.1736619

Cited by 24 publications

(28 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Similar conclusions were reached by Bardach et al (2020), who compared test-takers' reactions to three versions of an SJT. The results of their study did not indicate higher perceived fairness or job-relatedness for participants who took either of the two animated versions of the SJT compared to those who completed the text-based test.…”

Section: The Potential Of Video Technology In Reducing Construct-irrelevant Variancesupporting

confidence: 68%

See 1 more Smart Citation

Animated videos in assessment: comparing validity evidence from and test-takers’ reactions to an animated and a text-based situational judgment test

Karakolidis

O’Leary

Scully

2021

International Journal of Testing

View full text Add to dashboard Cite

The linguistic complexity of many text-based tests can be a source of construct-irrelevant variance, as test-takers' performance may be affected by factors that are beyond the focus of the assessment itself, such as reading comprehension skills. This experimental study examined the extent to which the use of animated videos, as opposed to written text, could (i) reduce construct-irrelevant variance attributed to language and reading skills and (ii) impact test-takers' reactions to a situational judgment test. The results indicated that the variance attributed to construct-irrelevant factors was lower by 9.5% in the animated version of the test. In addition, those who took the animated test perceived it to be more valid, fair, and enjoyable, than those who took the text-based test. They also rated the language used as less difficult to understand. The implications of these findings are discussed.

show abstract

Section: The Potential Of Video Technology In Reducing Construct-irrelevant Variancesupporting

confidence: 68%

“…One of the most relevant works in the field was undertaken by Bardach et al (2020) in the context of a study investigating ethnic bias. The authors compared three different forms of the same SJT (i. text-based, ii.…”

Section: The Potential Of Video Technology In Reducing Construct-irrelevant Variancementioning

confidence: 99%

Animated videos in assessment: comparing validity evidence from and test-takers’ reactions to an animated and a text-based situational judgment test

Karakolidis

O’Leary

Scully

2021

International Journal of Testing

View full text Add to dashboard Cite

show abstract

“…Accordingly, applicants were asked to rate the appropriateness of each of the options, from (1) appropriate to (4) inappropriate, in consideration of what a beginning teacher should do in the circumstances described in the scenario. The scoring key was developed using a hybrid approach, which combines two independently generated keys (see Bergman, Drasgow, Donovan, Henning, & Juraska, 2006). As such, concordance panels with subject matter experts (SMEs) in the field determined the initial scoring key.…”

Section: Sjtsmentioning

confidence: 99%

The selection gap in teacher education: Adverse effects of ethnicity, gender, and socio‐economic status on situational judgement test performance

Bardach

Rushby

Klassen

2021

Brit J of Edu Psychol

Self Cite

View full text Add to dashboard Cite

Background. Situational judgement tests (SJTs) measure non-cognitive attributes and have recently drawn attention as a selection method for initial teacher education programmes. To date, very little is known about adverse impact in teacher selection SJT performance.Aims. This study aimed to shed light on adverse effects of gender, ethnicity, and socioeconomic status (SES) on SJT scores, by exploring both main effects and interactions, and considering both overall SJT performance and separate SJT domain scores (mindset, emotion regulation, and conscientiousness).Sample. A total of 2,808 prospective teachers from the United Kingdom completed the SJTs as part of the initial stage of selection into a teacher education programme.Methods. In addition to SJT scores, the variables gender (female vs. male), ethnicity (majority group vs. minority group), and home SES background (higher SES status vs. lower SES status) were used in the analyses. Regression models and moderated regression models were employed.Results and conclusions. Results from the regression models revealed that gender effects (females scoring higher than males) were restricted to emotion regulation, while ethnicity effects (ethnic majority group members scoring higher than ethnic minority group members) emerged for SJT overall scores and all three domains. Moderated regression modelling results furthermore showed significant interactions (gender and ethnicity) for SJT overall scores and two domains. Considering the importance of reducing subgroup differences in selection test scores to ensure equal access to teacher education, this study's findings are a critical contribution. The partially differentiated results for overall vs. domain-specific scores point towards the promise of applying a domain-level perspective in research on teacher selection SJTs.Diversifying the teacher workforce has long been a concern of educational policy (e.g., Kirby, Berands, & Naftel, 1999). However, limited progress has been made to reach this goal (e.g., Albert Shanker Institute, 2015; OECD, 2016), as indicated by the relative scarcity ofThis is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.

show abstract

“…Unlike most measures of social intelligence or related ability constructs in organizational research (e.g., emotional intelligence or applied interpersonal skills assessed using SJTs), the Shapes Test uses animated, two-dimensional shapes to represent social interactions (Figure 1). These animations are similar in design to the appearance of early video games (e.g., Asteroids, Space Invaders, or Tetris) and are more a cost-effective alternative to creating higher-fidelity videos of human interactions (e.g., Bardach et al, 2021;Golubovich et al, 2017). We are specifically interested in demonstrating whether the animated shape task used in the Shapes Test yields weaker mean group score differences relative to a text-based SJT of social intelligence.…”

Section: Introductionmentioning

confidence: 99%

Using game‐like animations of geometric shapes to simulate social interactions: An evaluation of group score differences

Brown

Speer

Tenbrink

et al. 2022

Int J Selection Assessment

View full text Add to dashboard Cite

This study introduces a novel, game-like method for measuring social intelligence: the Social Shapes Test. Unlike other existing video or game-based tests, the Shapes Test uses animations of abstract shapes to represent social interactions.We explore demographic differences in Shapes Test scores compared to a written situational judgment test (SJT). Gender and race/ethnicity only had meaningful effects on written SJT, scores while no effects were found for Shapes Test scores. This pattern of results remained after controlling for general mental ability and English language exposure. We also found metric invariance between demographic groups for both tests. Our results demonstrate the potential for using animated shape tasks as an alternative to written SJTs when designing future game-based assessments.

show abstract

Using video- and text-based situational judgement tests for teacher selection: a quasi-experiment exploring the relations between test format, subgroup differences, and applicant reactions

Abstract: Using video-and text-based situational judgement tests for teacher selection: a quasi-experiment exploring the relations between test format, subgroup differences, and applicant reactions,

Cited by 24 publications

References 66 publications

Animated videos in assessment: comparing validity evidence from and test-takers’ reactions to an animated and a text-based situational judgment test

Animated videos in assessment: comparing validity evidence from and test-takers’ reactions to an animated and a text-based situational judgment test

The selection gap in teacher education: Adverse effects of ethnicity, gender, and socio‐economic status on situational judgement test performance

Using game‐like animations of geometric shapes to simulate social interactions: An evaluation of group score differences

Contact Info

Product

Resources

About