A state response measurement model for problem-solving process data

Abstract

In computer simulation-based interactive tasks, different people make different response processes to the same tasks, resulting in various action sequences. These sequences contain rich information, not only about respondents, but also about tasks. In this study, we propose a state response (SR) measurement model with a Bayesian approach for analyzing the process sequences, which assumes that each action made is determined by the individual's problem-solving ability and the easiness of the current problem state. This model is closer to reality compared with the action sub-model (referred to as DC model) of Chen's (2020) continuous-time dynamic choice (CTDC) measurement model that defines the easiness parameter only at the task level and ignores the task's process characteristics. The simulation study showed that the SR model performed well in parameter estimation. Moreover, the estimation accuracy of the SR model was quite similar to that of the DC model when state easiness parameters were equal within the task, but was much higher when within-task state easiness parameters were unequal. For the empirical data from the Program for International Student Assessment 2012, the SR model showed better model fit than the DC model. The estimates for state easiness parameters within each task were obviously different and made sense for characterizing task steps, further demonstrating the rationality of the proposed SR model.

Understanding the log file data from educational and psychological computer-based testing: A scoping review protocol

“…For instance, Scoular et al [ 23 ] and Drake et al [ 24 ] have demonstrated how student behavior from log file data, can be scored to assess performance in collaborative problem-solving tasks. And various models have been developed to estimate problem-solving abilities from action sequences [ 25 , 26 ] and student speed from action time [ 27 ].…”

External Correlates of Adult Digital Problem-Solving Process

“…Process data, on top of final scores, offer a wealth of information about individual differences, test-taking engagement, and the steps examinees take to reach their final response. Studies have demonstrated the utility of process data for a multitude of practical tasks: To start, process data can provide additional information on the measured proficiency or skills, allowing better measurement via process-incorporated scoring rules (Zhang et al, 2023) and process-based measurement models, which typically associate continuous latent proficiency (Chen, 2020; Han et al, 2022; LaMar, 2018; Liu et al, 2018; Xiao & Liu, 2024) or discrete latent skill mastery (Zhan & Qiao, 2022; Liang et al, 2022) with examinees’ choices of correct/incorrect subsequent actions, observed action subsequences, or sequence length. Furthermore, analyses of behavioral characteristics associated with successful/unsuccessful final performance (e.g., Gao, Cui, et al, 2022; Gao, Zhai, et al, 2022; Greiff et al, 2015; He & von Davier, 2016; Qiao & Jiao, 2018; Qiao et al, 2023; Ulitzsch et al, 2021, 2023) can inform test validation and automated scoring.…”