How Do Test Takers Interact With Simulation-Based Tasks? A Response-Time Perspective

Abstract

Many traditional educational assessments use multiple-choice items and constructed-response items to measure fundamental skills. Virtual performance assessments, such as game- or simulation-based assessments, are designed recently in the field of educational measurement to measure more integrated skills through the test takers’ interactive behaviors within an assessment in a virtual environment. This paper presents a systematic timing study based on data collected from a simulation-based task designed recently at Educational Testing Service. The study is intended to understand the response times in complex simulation-based tasks so as to shed light on possible ways of leveraging response time information in designing, assembling, and scoring of simulation-based tasks. To achieve this objective, a series of five analyses were conducted to first understand the statistical properties of the timing data, and then investigate the relationship between the timing patterns and the test takers’ performance on the items/task, demographics, motivation level, personality, and test-taking behaviors through use of different statistical approaches. We found that the five analyses complemented each other and revealed different useful timing aspects of this test-taker sample’s behavioral features in the simulation-based task. The findings were also compared with notable existing results in the literature related to timing data.

Identifying patterns of students' performance on simulated inquiry tasks using PISA 2015 log‐file data

“…To identify profiles of inquiry performance, we combined item accuracy with students' behavioral actions during the inquiry process and the time they spent to solve the tasks. While the item accuracy can explain what students have achieved, the behavioral actions and time‐on‐task may contribute to a deeper understanding of the process that underlies the overall achievement (Y. H. Lee et al, 2019; Peffer & Ramezani, 2019). Studies have demonstrated that the frequency of specific behaviors, such as the number of actions and trials students conducted, plays an important role in understanding task success (e.g., Greiff et al, 2015; Greiff et al, 2018).…”

“…Some evidence suggests that the time students spend on a task is positively correlated with task success (e.g., Y. Lee, 2018; Scherer, Greiff, & Hautamäki, 2015) and the effort taken to solve the task (e.g., Kristen, 2014; Ventura & Shute, 2013). The time spent on a task consequently represents the person‐task interactions and can provide information about the response process, behaviors, and motivation during the assessment beyond the correctness of responses (Goldhammer et al, 2014; Y. H. Lee, Hao, Man, & Ou, 2019).…”

What Technology Can and Cannot Do to Support Assessment of Non-cognitive Skills

“…Log files of online game-based assessments include time and event information that can be used to track participants’ collaboration during the game (Hao et al, 2016; Hao and Mislevy, 2018). Process data may provide insight into responses that would not be possible without technology, and analyzing such data can support assessment validation (Lee et al, 2019).…”

Applying Evidence-Centered Design to Measure Psychological Resilience: The Development and Preliminary Validation of a Novel Simulation-Based Assessment Methodology

“…Well-designed interactive technology (e.g., games, simulations), also referred to as Virtual Performance assessments (VPAs) show promise as a means to measure complex non-cognitive constructs including resilience, and have been referred to as the next-generation of assessment ( de-Juan-Ripoll et al, 2018 ; Hao and Mislevy, 2018 ; Lee et al, 2019 ). A recent scoping review found interactive technology can deliver effective interventions to increase resilience ( Pusey et al, 2020 ).…”

“…They were chosen as variables of interest as they give measurable, unambiguous and objective outcomes of performance level. A response-time in complex simulation-based tasks is generally suggested to be in scoring ( Lee et al, 2019 ). Speed was defined as an arbitrary digital miles per hour, and collisions were measured as the number of times a participant’s vehicle collided with external objects.…”