Animated videos in assessment: comparing validity evidence from and test-takers’ reactions to an animated and a text-based situational judgment test

Abstract

The linguistic complexity of many text-based tests can be a source of construct-irrelevant variance, as test-takers' performance may be affected by factors that are beyond the focus of the assessment itself, such as reading comprehension skills. This experimental study examined the extent to which the use of animated videos, as opposed to written text, could (i) reduce construct-irrelevant variance attributed to language and reading skills and (ii) impact test-takers' reactions to a situational judgment test. The results indicated that the variance attributed to construct-irrelevant factors was lower by 9.5% in the animated version of the test. In addition, those who took the animated test perceived it to be more valid, fair, and enjoyable, than those who took the text-based test. They also rated the language used as less difficult to understand. The implications of these findings are discussed.

Situated and dynamic versus declarative and static forms of pedagogical content knowledge: An evaluation of the differences in test reactions and performance

“…In terms of the nature of the test (RQ2), the participants believed that the video‐based test was more authentic and related to their actual job demands than the text‐based test. This finding is consistent with the literature on the affordances of video as a means of offering authentic contexts for teachers to consider (Sherin, 2004) and with prior studies comparing video‐based and text‐based teacher competence assessments (Karakolidis et al, 2021). In relation to test experiences (RQ3), the teachers perceived positive affective experiences (i.e., engagement, immersion, effort) in the video‐based test but also reported experiencing higher levels of anxiety and cognitive demand.…”

“…In relation to test experiences (RQ3), the teachers perceived positive affective experiences (i.e., engagement, immersion, effort) in the video‐based test but also reported experiencing higher levels of anxiety and cognitive demand. The teachers' positive affective experiences align with the literature on the affordances of video as an immersive and engaging medium (Goldman, 2007; Seidel et al, 2011) and with the findings of teacher assessment studies (Bardach et al, 2021; Karakolidis et al, 2021). Video‐based teacher competence test designers have argued that video‐based instruments need to be well balanced between authenticity and the cognitive demand they impose on teachers (Codreanu et al, 2020; Keppens et al, 2019; Seidel et al, 2010).…”

“…Significantly, the participants' SJT scores were positively predicted by engagement in the video without text format. More recently, Karakolidis et al (2021) conducted an experimental study ( N = 129) in Ireland and Greece to examine preservice teachers' reactions to different formats of SJT. Those who took the animated video version ( N = 66) considered the SJT to be significantly more fair and more relevant to the teaching profession than those who took the text‐based version ( N = 63).…”

Using game‐like animations of geometric shapes to simulate social interactions: An evaluation of group score differences

“…Generally speaking, some scholars suggest that using video stimuli in place of text may help minimize group score differences (Chan & Schmitt, 1997). This is theorized to be caused by minimizing construct-irrelevant variances such as reading ability or cognitive load (Karakolidis et al, 2021). However, most of the research on video elements has focused on tests involving human actors or animated characters (Golubovich et al, 2017).…”

The Social Shapes Test as a Self-Administered, Online Measure of Social Intelligence: Two Studies with Typically Developing Adults and Adults with Autism Spectrum Disorder

“…This may help researchers recruit larger samples or may make participating in research studies more accessible to potential participants. Based on findings from recent research, the use of animation in the SST may also create a more engaging and enjoyable experience for participants relative to text-based assessments (Karakolidis et al, 2021 ).…”